Technical  Report  1237 


Evaluating  the  0*NET  Occupational  Analysis  System 
for  Army  Competency  Development 


Teresa  L.  Russell,  Andrea  Sinclair, 

Jesse  Erdheim,  and  Michael  Ingerick 

Human  Resources  Research  Organization 

Kimberly  Owens 

U.S.  Army  Research  Institute 

Norman  Peterson  and  Kenneth  Pearlman 

Independent  Consultants 


July  2008 


United  States  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences 


Approved  for  public  release;  distribution  is  unlimited. 


20080811  033 


U.S.  Army  Research  Institute 

for  the  Behavioral  and  Social  Sciences 

A  Directorate  of  the  Department  of  the  Army 
Deputy  Chief  of  Staff,  G1 

Authorized  and  approved  for  distribution: 


MICHELLE  SAMS,  PhD. 
Director 


Technical  review  by 


Elizabeth  Brady,  U.S.  Army  Research  Institute 
Richard  Hoffman,  U.S.  Army  Research  Institute 


NOTICES 


DISTRIBUTION:  Primary  distribution  of  this  Technical  Report  has  been  made  by  ARI. 
Please  address  correspondence  concerning  distribution  of  reports  to:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  Attn:  DAPE-ARI-ZXM, 

2511  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926. 

FINAL  DISPOSITION:  This  Technical  Report  may  be  destroyed  when  it  is  no  longer 
needed.  Please  do  not  return  it  to  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 


NOTE:  The  findings  in  this  Technical  Report  are  not  to  be  construed  as  an  official 
Department  of  the  Army  position,  unless  so  designated  by  other  authorized  documents. 


REPORT  DOCUMENTATION  PAGE 

1  REPORT  DATE  (dd-mm-yy) 

July  2008 

2.  REPORT  TYPE 

Final 

3.  DATES  COVERED  (from.  .  .  to) 

June  2007  -  April  2008 

4.  TITLE  AND  SUBTITLE 

Evaluating  the  O'NET  Occupational  Analysis  System  for  Army 

5a.  CONTRACT  OR  GRANT  NUMBER 

DASW01 -03-D-001 5;  Delivery  Order  0039 

Competency  Development 

5b.  PROGRAM  ELEMENT  NUMBER 

665803 

6.  AUTHOR(S) 

Teresa  L.  Russell,  Andrea  Sinclair,  Jesse  Erdheim,  Michael 
lngerick(Human  Resources  Research  Organization);  Kimberly 
Owens,  (U.S.  Army  Research  Institute);  Norman  Peterson,  and 
Kenneth  Pearlman 

5c.  PROJECT  NUMBER 

D730 

5d.  TASK  NUMBER 

319 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

5e.  WORK  UNIT  NUMBER 

Human  Resources  Research  Organization 

66  Canal  Center  Plaza,  Suite  400 

Alexandria,  Virginia  22314 

8.  PERFORMING  ORGANIZATION  REPORT  NUMBER 
FR-08-65 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 

10.  MONITOR  ACRONYM 

ARI 

Sciences 

251 1  Jefferson  Davis  Highway 
Arlington,  VA  22202-3926 

11.  MONITOR  REPORT  NUMBER 

Technical  Report  1237 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

13.  SUPPLEMENTARY  NOTES 

Contracting  Officer's  Technical  Representative:  Kimberly  Owens.  Subject  Matter  POC:  Teresa  L.  Russell 

14.  ABSTRACT  (Maximum  200  words): 

The  present  evaluation  focused  primarily  on  the  usefulness  of  the  0*NET  system  for  Army  occupational  analysis  for 
selection  and  classification  purposes.  The  evaluation  focused  on  the  appropriateness  of  0*NET  descriptors  that  would 
typically  be  used  in  an  Army  occupation  analysis  for  selection  and  classification  purposes:  abilities,  skills,  generalized 
work  activities  [GW As],  and  work  context.  Four  civilian  and  four  officer  occupations  were  selected  for  this  research.  The 
objective  was  to  produce  data  for  the  military  occupations  that  could  be  compared  to  civilian  0*NET  data.  Therefore,  it 
was  important  to  follow  processes  currently  used  by  0*NET  for  data  collection.  In  effect,  this  meant  collecting 
information  on  occupational  tasks,  abilities  skills,  GWAs,  and  work  context  from  Army  Subject  Matter  Experts  (SMEs) 
and  collecting  ability  and  skill  ratings  using  trained  analysts.  The  results  showed  that  Army  SMEs  as  well  as  other 
types  of  analysts  could  make  reliable  ratings  on  the  0*NET  descriptors.  Potential  uses  for  the  0*NET  descriptors  are 
discussed. 

15.  SUBJECT  TERMS 

Competency;  0*NET,  job  analysis;  officers 

SECURITY  CLASSIFICATION  OF 

19.  LIMITATION  OF 

20.  NUMBER 

21.  RESPONSIBLE  PERSON 

ABSTRACT 

OF  PAGES 

(Name  and  Telephone  Number) 

16.  REPORT 

17.  ABSTRACT 

18.  THIS  PAGE 

Unlimited 

174 

Ellen  Kinzer 

Unclassified 

Unclassified 

Unclassified 

Technical  Publications  Specialist 
(703)  602-8047 

Standard  Form  298 


11 


Technical  Report  1237 


Evaluating  the  0*NET  Occupational  Analysis  System  for 
Army  Competency  Development 


Teresa  L.  Russell,  Andrea  Sinclair, 
Jesse  Erdheim,  and  Michael  Ingerick 

Human  Resources  Research  Organization 

Kimberly  Owens 

U.S.  Army  Research  Institute 

Norman  Peterson  and  Kenneth  Pearlman 

Independent  Consultants 


Personnel  Assessment  Research  Unit 
Michael  G.  Rumsey,  Chief 


U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
2511  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926 


July  2008 


Army  Project  Number  Personnel  and  Training 

665803D730  Analysis  Activities 

Approved  for  public  release;  distribution  is  unlimited. 


in 


i 


IV 


EVALUATING  THE  0*NET  OCCUPATIONAL  ANALYSIS  SYSTEM  FOR 
ARMY  COMPETENCY  DEVELOPMENT 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  Quadrennial  Defense  Review  (2006)  called  for  an  “in  depth  study  of  the 
competencies  U.S.  forces  require  and  the  performance  standards  to  which  they  must  be 
developed”  (p.  80).  The  Department  of  Labor’s  (DOL)  occupational  information  database 
(0*NET;  Peterson,  Mumford,  Borman,  Jeanneret,  &  Fleishman,  1 999)  provides  a  logical  starting 
point  for  identifying  the  structure  and  composition  of  an  occupational  database  that  will  meet  the 
Army’s  needs.  Indeed,  the  Army  has  previously  commissioned  a  brief  concept  paper  on  just  this 
issue  (Russell,  Mumford,  &  Peterson,  1 996)  as  well  as  a  more  far-reaching  review  of  the  0*NET 
and  occupational  analysis  in  general  (Committee  on  Techniques  for  the  Enhancement  of  Human 
Performance:  Occupational  Analysis,  1999).  Continuing  this  reasoned  approach,  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  initiated  this  project  to  evaluate 
the  utility  of  0*NET  for  describing  Army  occupation  competencies. 

Procedure: 

The  present  evaluation  focused  primarily  on  the  usefulness  of  the  0*NET  system  for 
Army  occupational  analysis  for  selection  and  classification  purposes.  The  0*NET,  which  is 
organized  around  a  content  model,  contains  a  wealth  of  data,  and  evaluating  all  of  0*NET  for  all 
possible  human  resource  needs  would  be  beyond  the  scope  of  this  effort.  The  evaluation  focused 
on  the  appropriateness  of  0*NET  descriptors  that  would  typically  be  used  in  an  Army 
occupation  analysis  for  selection  and  classification  purposes.  These  included  abilities,  skills, 
Generalized  Work  Activities  [GW As],  and  work  context. 

The  evaluation  was  designed  to  address  the  following  six  questions: 

1 .  Are  Army  occupations  rated  reliably  using  0*NET  rating  scales? 

2.  Do  ratings  on  0*NET  rating  scales  differentiate  Army  occupations? 

3.  How  well  do  Subject  Matter  Expert  (SME)  and  analyst  ratings  agree? 

4.  Are  ratings  on  Army  occupations  similar  in  quality  to  ratings  on  civilian  occupations? 

5.  Are  Army  occupational  profiles  similar  to  those  for  their  civilian  counterparts? 

6.  How  well  do  0*NET’s  work  requirements  descriptors,  particularly  generalized  work 
activities  (GW As),  cover  Army  job  requirements? 

Addressing  the  first  five  questions  required  collection  of  ratings  on  the  selected  0*NET 
descriptors — abilities,  skills,  GW  As,  and  work  context — for  target  Army  occupations.  Four 
civilian  and  four  officer  occupations  were  selected  for  this  research.  The  objective  was  to 
produce  data  for  the  military  occupations  that  could  be  compared  to  civilian  0*NET  data. 
Therefore,  it  was  important  to  follow  processes  currently  used  by  0*NET  for  data  collection.  In 


effect,  this  meant  collecting  information  on  occupational  tasks,  abilities  skills,  GWAs,  and  work 
context  from  Army  Subject  Matter  Experts  (SMEs)  and  collecting  ability  and  skill  ratings  using 
trained  analysts. 

SMEs  were  non-commissioned  officers  (NCOs)  or  officers  with  several  years  of 
experience  in  the  Army  and  their  occupations,  and  who  thus  had  first-hand  observation  and 
experience  of  the  focal  occupations.  Prior  to  SME  workshops,  we  drafted  lists  of  the  major  duties 
and  Key  Work  Activities  (KWAs)  for  their  occupation.  In  the  workshops,  SMEs  reviewed  and 
finalized  the  occupation-specific  materials  and  then  rated  0*NET  abilities,  skills,  GWAs,  and 
work  context  variables. 

Two  types  of  analysts  were  included  in  this  research — 0*NET  analysts  and  project 
analysts.  0*NET  analysts  were  individuals  who  currently  make  operational  ability  and  skill 
ratings  for  0*NET.  Project  analysts  were  HumRRO  and  ARI  personnel  who  had  prior 
experience  with  Army  occupational  analysis.  0*NET  trainers  allowed  us  to  collect  our  analyst 
data  within  the  context  of  scheduled  training  and  data  collection  for  0*NET.  0*NET  and  project 
analysts  attended  a  2  Vi  day  training  session  on  rating  abilities  and  skills.  KWAs,  GWAs,  and 
work  context  ratings  from  the  SME  workshops  served  as  stimulus  materials  for  analysts  to  use  in 
making  their  ratings. 

The  sixth  question  required  somewhat  different  data.  If  0*NET  GWAs  appeared  to  be 
related  to  or  “cover”  all  or  many  of  the  Army  job  requirements  (called  Major  Duties  (MDs))  that 
are  common  across  occupations,  then  they  would  potentially  be  quite  useful  for  providing  a 
higher-order  description  of  Army  occupations  (i.e.,  one  that  would  encompass  the  major  duties 
performed  in  the  Army).  If  not,  additions  or  revisions  to  the  GWAs  might  be  needed.  In 
preparation  for  this  assessment  of  coverage,  we  developed  a  major  duties  list  based  on  prior 
research  and  obtained  feedback  on  it  from  Army  SMEs.  To  make  this  evaluation,  we  asked  non- 
incumbent  raters  to  rate  the  degree  to  which  each  MD  was  “covered”  by  each  GWA,  with  no 
occupation  specifically  identified  (i.e.,  the  MD  might  be  imagined  to  be  a  part  of  any  of  several 
occupations,  no  particular  occupation  was  implicated). 

Findings: 

The  0*NET  analyst  training  package  appears  to  work  fairly  well.  Results  presented  in 
Chapter  III  suggest  that  trained  analysts  make  reliable  ability  ratings.  With  some  training,  SMEs 
might  do  so  as  well,  albeit  SMEs  are  likely  to  take  issue  with  the  lack  of  Army-specificity  in  the 
anchors.  We  also  found  that  trained  project  and  0*NET  analysts  were  able  to  rate  the  abilities 
with  about  the  same  level  of  agreement  and  reliability  observed  for  0*NET  analyst  ratings  of 
civilian  occupations. 

The  ability  level  ratings  show  appropriate  convergent  and  divergent  validity  for  enlisted 
occupations.  That  is,  the  correlation  between  occupations  on  abilities  was  low  relative  to  the 
correlations  among  different  rater  types.  This  suggests  that  the  abilities  would  provide  a  useful 
basis  for  distinguishing  jobs  for  classification  purposes.  We  did,  however,  find  that  project 
analysts  drew  more  distinctions  than  0*NET  analysts.  This  suggests  that  trained  analysts  should 
be  ones  who  know  something  about  Army  jobs.  We  also  found  that  the  abilities  were  less 
distinguishing  for  officer  occupations,  but  it  is  possible  that  officer  occupations  are  truly  more 
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similar  to  each  other  than  enlisted  ones  are,  since  officers  have  common  responsibilities 
regardless  of  occupation. 

Finally,  we  applied  the  multi-trait,  multi-method  approach  to  assess  the 
convergence/divergence  of  0*NET  ability  data  for  Army  occupations  with  0*NET  ability  data 
for  civilian  counterpart  occupations  in  the  0*NET  database.  We  found  that  ability  ratings  for 
Army  occupations  correlated  most  highly  with  ability  ratings  for  their  civilian  counterparts 
(mean  r  =  .73);  Army  occupation  ability  ratings  did  not  correlate  as  highly  ability  ratings  for 
non-counterpart  civilian  occupations  (mean  r  =  .41). 

All  rater  types  rated  skills  very  reliably,  and  there  were  no  large  differences  in  the 
magnitude  of  ratings  made  by  different  rater  types  on  skills.  When  all  eight  occupations  were 
considered,  the  skill  ratings  showed  differentiation  across  occupations.  That  differentiation 
diminished  when  we  looked  within  enlisted  and  officer  occupations,  most  of  the  differentiation 
probably  resulting  from  differences  between  officer  and  enlisted  occupation  skill  level 
requirements.  Skill  level  means  were  considerably  lower,  on  average,  for  entry-level  enlisted 
jobs  than  for  entry-level  officers;  it  makes  sense  that  higher  levels  of  skills  are  more  likely  to 
come  into  play  as  Soldiers  move  up  the  ranks.  As  with  the  abilities,  Army  occupations  were 
more  highly  correlated  with  counterpart  civilian  occupations  on  skills  than  non-counterpart 
occupations,  another  piece  of  evidence  supporting  skill  rating  validity. 

While  the  empirical  data  for  the  work  context  variables  were  strong  (e.g.,  SMEs  rated 
them  reliably,  they  differentiated  jobs),  verbal  reports  from  the  SME  workshops  suggested  areas 
in  which  the  work  context  descriptors  might  be  improved.  In  particular,  the  work  context 
descriptors  could  be  helpful  in  describing  in-garrison  and  deployment  contexts — which  were  a 
frequent  point  of  discussion  in  the  workshops.  Based  on  other  comments  from  SMEs  in  the 
workshops,  possible  additions  to  the  descriptor  set  include  descriptions  dealing  with  the 
following:  work  pace;  sleep  deprivation;  communication  with  indigenous  people,  host  nation 
counterparts,  peers,  and  supervisors;  travel,  overseas  travel,  and  extended  time  away  from  home; 
and  lifting  heavy  weight. 

The  data  suggest  that  if  the  Army  were  to  collect  GWA  ratings  from  about  1 5  to  18 
SMEs  for  each  occupation,  a  recommended  multi-rater  reliability  of  .80  would  be  achieved.  The 
data  also  suggest  that  those  GWA  ratings  would  differentiate  occupations.  The  convergent 
correlations  between  Army  occupations  and  SOC  counterparts,  in  conjunction  with  the  divergent 
correlations  between  Army  and  non-counterpart  occupations,  provide  additional  support  for  the 
convergent  and  divergent  validity  of  the  GWA  ratings  made  by  SMEs. 

The  task  of  assessing  the  coverage  of  MDs  by  GW  As  showed  that  most  MDs  were 
covered  by  GWAs.  Based  on  the  criteria  we  imposed,  67  of  the  87  MDs  had  “full”  coverage,  five 
had  “High  Partial”  coverage,  1 1  had  “low  partial”  coverage,  and  four  were  not  covered. 
Interestingly,  the  15  with  low  partial  or  no  coverage  were  not  necessarily  Army-specific.  We 
grouped  them  as  follows:  trades-related  (e.g.,  install,  maintain,  and  repair  pipe  assemblies), 
hazard/combat  related  (e.g.,  fire  direct  fire  weapons),  team-related  (e.g.,  help  peers  and 
individual  team  members),  and  miscellaneous  (e.g.,  prepare  food  and  beverages). 
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Utilization  and  Dissemination  of  Findings: 

The  results  of  this  study  suggest  that  the  rapid  development  and  implementation  of  an 
Army-specific  occupational  analysis  system  is  practicable  using  0*NET  descriptors  as  a 
foundation.  Recommendations  include: 

•  Incorporate  the  ability  and  skill  domains  of  0*NET  “as  is;” 

•  Make  improvements  by  adding  key  descriptors  to  the  0*NET  GWAs  and  work  context 
domains,  as  described  in  the  final  chapter; 

•  Develop  and  refine  and  Army-wide  domain  of  Key  Work  Activities,  organized  into 
higher-order  Major  Duties; 

•  Use  a  cadre  of  trained  analysts  to  make  the  ability  and  skill  ratings  (the  standard  0*NET 
training  given  to  either  experienced  Army  SMEs  or  Army  scientists  would  be  sufficient); 

•  Use  samples  of  1 5  to  30  SMEs  to  make  the  GWA  and  work  context  ratings  for  each 
Army  occupation. 

This  system  would  provide  an  extremely  useful  “common  language”  occupational 
analysis  system  for  the  Army  with  strong  links  to  the  civilian  occupational  database.  Such  a  link 
has  obvious  benefits  for  recruitment  and  rapid  mobilization  efforts.  Selection  and  classification 
applications  should  flow  directly  from  such  a  system  as  described  in  Campbell  et  al.  2006  and 
training  and  development  needs  could  be  met  with  a  linkage  of  specific  occupational  tasks  to 
KWAs  and,  hence,  to  Major  Duties,  as  we  briefly  described  in  Chapter  II.  Opportunities  for 
efficiencies  in  training  and  development  applications  would  seem  much  easier  to  identify  with 
such  a  system,  as  would  the  definition  and  development  of  job  performance  criteria  linked  to 
KWAs  and  MDs. 


viii 


EVALUATING  THE  0*NET  OCCUPATIONAL  ANALYSIS  SYSTEM  FOR 
ARMY  COMPETENCY  DEVELOPMENT 


CONTENTS 


Page 

I.  Introduction  and  Purpose . 1 

Purpose  of  the  Project . 2 

Relevant  History . 2 

0*NET . 2 

Army  Occupational  Analysis . 3 

Overview  of  Approach . 4 

Overview  of  Report . 5 

II.  Method  of  Describing  Army  Occupations  Using  0*NET . 7 

SME  Data  Collection  Methods . 8 

Identification  of  Target  Occupations . 8 

Development  of  Key  Work  Activities  and  Major  Duties . 8 

SME  Workshops . 9 

Analyst  Data  Collection . 16 

Stimulus  Materials  for  the  Ratings . 16 

Overview  of  the  Training . 18 

Army-Wide  and  MOS-Specific  Tasks . 19 

Rating  Assignments  and  Results . 20 

III.  Is  0*NET  Sufficient? . 21 

Are  Army  Occupations  Rated  Reliably  Using  0*NET  Rating  Scales? . 21 

Abilities  and  Skills . 21 

Work  Context . 22 

Generalized  Work  Activities . 26 

Do  Ratings  on  0*NET  Rating  Scales  Differentiate  Army  Occupations? . 26 

Convergent  Correlations  based  on  Ability  and  Skill  Level  Ratings . 27 

Divergent  Correlations  Based  on  Ability  and  Skill  Level  Ratings . 27 

Ability  and  Skill  Level  Divergence  and  Convergence  Summary . 31 

Convergence  and  Divergence  of  WCs  and  GW  As . 31 

How  Well  Do  SME  and  Analyst  Ratings  Agree? . 33 

Are  Ability  Ratings  for  Army  Occupations  Similar  in  Quality  to  Ratings  for  Civilian 

Occupations? . 38 

Ability  Flags . 38 

Descriptive  Statistics  and  Reliabilities  for  Abilities . 39 

Do  the  Score  Profiles  for  Army  Occupations  Look  Like  Those  for  Similar  Civilian 

Occupations? . 39 

Summary . 44 


Do  raters  rate  Army  occupations  reliably  using  0*NET  rating  scales? . 44 

Do  ratings  on  0*NET  rating  scales  differentiate  Army  occupations? . 45 

How  well  do  SME  and  analyst  ratings  agree? . 46 

Are  ratings  on  Army  occupations  similar  in  quality  to  ratings  on  civilian 

occupations? . 46 

Are  Army  occupation  profiles  similar  to  those  for  their  civilian  counterparts? . 46 

IV.  Analysis  of  Fit  Between  Major  Duties  and  Generalized  Work  Activities . 47 

Purpose . 47 

MD  x  GWA  Exercise  Method . 47 

Results . 49 

Reliability . 49 

Descriptive  Statistics . 49 

V.  Summary  and  Discussion . 55 

Abilities . 56 

Summary  of  Ability  Results . 56 

Ability  Descriptor  Recommendations . . 57 

Skills . 57 

Summary  of  Skill  Results . . . 57 

Skill  Descriptor  Recommendations . 57 

Work  Context . . 58 

Summary  of  Work  Context  Results . 58 

Work  Context  Descriptor  Recommendations . 58 

GWAs . 60 

Summary  of  GWA  Results . 60 

GWA  Descriptor  Recommendations . 60 

A  Vision  for  the  Future . 61 

VI.  References . 63 


LIST  OF  TABLES 

Table  1.  SME  Workshop  Participants  by  Level  and  Occupation . 14 

Table  2.  Descriptive  Statistics  for  Key  Work  Activity  Ratings . 14 

Table  3.  Intraclass  Correlations  (ICCs)  by  Occupation  for  KWA  Ratings . 16 

Table  4.  ICCs  for  Ability  Importance  and  Level  Ratings  by  Occupation  and  Rater  Type . 23 

Table  5.  ICCs  for  Skill  Importance  and  Level  Ratings  by  Occupation  and  Rater  Type . 24 

Table  6.  ICCs  by  Occupation  for  Work  Context  Ratings . 25 

Table  7.  ICCs  by  Occupation  for  Generalized  Work  Activity  Ratings . 26 

Table  8.  Ability  Level  Correlations  by  Occupation  and  Rater  Type . 28 

Table  9.  Skill  Level  Correlations  by  Occupation  and  Rater  Type . 29 

Table  10.  Mean  Convergent  and  Divergent  Correlations  for  Ability  and  Skill  Level  Ratings  ....  30 
Table  1 1 .  Work  Context  Correlations  by  Occupation . 32 


Table  12.  GWA  Level  Correlations  by  Occupation . 32 

Table  13.  Mean  Convergent  and  Divergent  Correlations  for  WC  and  GWA  Level  Ratings . 32 

Table  14.  Summary  Descriptive  Statistics  on  Ability  and  Skill  Scales  Across  Occupations  by 

Rater  Type . 34 

Table  15.  Effect  Sizes  for  Rater  Types  on  Ability  Level  Scales . 35 

Table  16.  Effect  Sizes  for  Rater  Types  on  Skill  Level  Scales . 37 

Table  17.  Comparability  of  Military  and  Civilian  Ability  Ratings  in  Terms  of  Flags . 41 

Table  18.  Comparison  of  Civilian  and  Army  Descriptive  Statistics  for  Ability  Ratings  made  by 

0*NET  Analysts . 42 

Table  19.  Comparability  of  Civilian  and  Army  Reliabilities  for  Ability  Ratings  made  by  0*NET 

Analysts . 42 

Table  20.  Standard  Occupational  Classification  (SOC)  Counterparts  for  Army  Target 

Occupations . 42 

Table  21 .  Correlations  of  Ability  Level  Profile  Means  for  Target  Army  and  SOC  Occupations  43 
Table  22.  Summary  of  Intercorrelations  Among  Army  and  Selected  SOC  Counterpart 

Occupations . 44 

Table  23.  Reliability  estimates  for  GWA  Extent-of-Coverage  Ratings . 49 

Table  24.  Summary  of  Ratings  of  "Coverage"  By  GWAs  of  Army  Major  Duties  (By  Number  of 
Average  Ratings  >  .99) . 5 1 

LIST  OF  FIGURES 

Figure  1.  Block  1  agenda . 10 

Figure  2.  Block  2  agenda . 1 1 

Figure  3.  KWA  rating  scales . 12 

Figure  4.  Example  work  context  item . 1 3 

Figure  5.  Example  0*NET  importance  and  level  rating  scales . 13 

LIST  OF  APPENDICES 

A.  The  Feasibility  of  Using  0*NET  to  Study  Skill  Changes  (Tsacoumis,  2007) 

B.  MDs  Assigned  to  GWA  Factors 

C.  Major  Duty  and  GWA  Definitions 

D.  Descriptive  Statistics  for  MD-GWA  Ratings 

E.  Summary  of  Ratings  of  "Coverage"  By  GWAs  of  Army  Major  Duties  (By  Major  Duty 
Number) 


xi 


xii 


EVALUATING  THE  0*NET  OCCUPATIONAL  ANALYSIS  SYSTEM  FOR  ARMY 

COMPETENCY  DEVELOPMENT 


I.  Introduction  and  Purpose 

“To  compete  effectively  with  the  civilian  sector  for  highly-qualified  personnel  to  build 
the  Total  Force,  the  Department  [of  Defense]  must  possess  both  a  modem  human 
capital  strategy  and  the  authorities  required  to  recruit,  shape,  and  sustain  the  force  it 
needs.  The  new  Human  Capital  Strategy  focuses  on  developing  the  right  mix  of  people 
and  skills  across  the  Total  Force.  The  Department’s  Human  Capital  Strategy  may  be 
considered  ‘competency- focused’  and  ‘performance-based.’  It  is  based  on  an  indepth 
study  of  the  competencies  U.S.  forces  require  and  the  performance  standards  to  which 
they  must  be  developed.  Each  of  the  Military  Departments  will  map  the  array  of 
competencies  and  performance  criteria  that  constitute  its  forces  and  also  evaluate  and 
improve  personnel  development  processes  to  achieve  those  standards.  Advancements, 
awards  and  compensation  may  then  be  linked  to  an  individual’s  performance  rather  than 
to  longevity  or  time-in-grade.  This  will  better  align  incentives  to  outputs  and  reward 
excellence”  [ Quadrennial  Defense  Review  Report  (QDR),  2006,  p.  80]. 

The  QDR  (2006)  presents  challenges  and  opportunities  for  Army  occupational  analysis.  It 
presents  an  opportunity  to  develop  a  centralized,  digitized  occupational  information  system. 
Ideally,  the  system  would  be  designed  to  provide  occupational  information  that  is  easily 
accessible  and  well  suited  to  the  Army’s  human  resource  management  needs.  One  challenge  has 
to  do  with  the  intent  and  language  of  the  QDR.  While  it  implies  that  the  system  would  be  used 
for  training/development  and  compensation  purposes,  these  and  other  uses  are  not  fully 
articulated.  The  QDR  uses  the  terms  “competency- focused”  and  “performance-based”  without 
defining  them.  There  are  several  ways  to  interpret  those  terms;  a  wide  range  of  work-  and 
worker-oriented  descriptors  have  been  referred  to  as  competencies  in  the  literature  (Sackett  & 
Laczo,  2003).'  Another  challenge  is  that  designing  the  information  system  will  require  research 
and  innovation,  and  populating  it  will  require  time  and  effort.  These  challenges  are 
surmountable,  and  meeting  them  could  result  in  an  Army  occupational  information  database  that 
is  well  worth  the  effort  (Campbell,  McCloy,  Morton,  Pearlman,  Peterson,  Rounds,  &  Ingerick, 
2006). 


The  Department  of  Labor’s  (DOL)  occupational  information  database  (0*NET;  Peterson, 
Mumford,  Borman,  Jeanneret,  &  Fleishman,  1999)  provides  a  logical  starting  point  for 
identifying  the  structure  and  composition  of  an  occupational  database  that  will  meet  the  Army’s 
needs.  Indeed,  the  Army  has  previously  commissioned  a  brief  concept  paper  on  just  this  issue 
(Russell,  Mumford,  &  Peterson,  1996)  as  well  as  a  more  far-reaching  review  of  the  0*NET  and 
occupational  analysis  in  general  (Committee  on  Techniques  for  the  Enhancement  of  Human 
Performance:  Occupational  Analysis,  1999).  Continuing  this  reasoned  approach,  the  U.S.  Army 


For  the  purposes  of  this  project,  we  assume  that  “competency”  is  a  broad  term  that  refers  to  many  types  of  job  information 
Where  possible,  we  use  the  more  specific  terms  “worker-oriented”  characteristics  (e.g.,  occupation-specific  knowledge,  abilities) 
and  “work-oriented”  descriptors  (e.g.,  performance  requirements,  work/job  context  and  machine-tools-equipment-technology). 


1 


Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  initiated  a  contract  to  the  Human 
Resources  Research  Organization  (HumRRO)  to  evaluate  the  utility  of  0*NET  for  describing 
Army  occupation  competencies. 


Purpose  of  the  Project 

The  primary  purpose  of  the  project  was  to  evaluate  the  usefulness  of  the  0*NET  system 
for  Army  occupational  analysis.  This  is  not  a  simple  question,  and  it  does  not  have  a  simple 
“yes”  or  “no”  answer.  Our  objective  was  to  identify  descriptors  from  0*NET  that  are  useful  “as 
is”  to  the  Army  and  those  that  should  be  modified,  supplemented,  or  deleted  altogether. 

The  evaluation  was  intended  to  speak  to  the  QDR  dictum  that  “each  of  the  Military 
Departments  will  map  the  array  of  competencies  and  performance  criteria  that  constitute  its 
forces  and  also  evaluate  and  improve  personnel  development  processes  to  achieve  those 
standards”  (p.  80).  This  poses  the  question,  would  0*NET  be  sufficient,  by  itself,  in  describing 
Army  occupations?  If  not,  how  is  it  insufficient  and  what  are  the  remedies?  Data  collections  and 
analyses  were  conducted  to  address  these  key  questions. 

Relevant  History 


0*JV£T 

The  U.S.  Department  of  Labor  (DOL)  created  0*NET  in  the  1990s  to  replace  its 
predecessor,  the  Dictionary  of  Occupational  Titles  (DOT).  The  DOT  had  originally  been 
developed  during  the  Great  Depression  as  a  way  of  providing  standardized  occupational 
information  to  public  employment  services — information  that  could  be  used  to  match  occupation 
applicants  to  jobs  (Dunnette,  1999).  By  1945,  the  DOT  contained  17,500  jobs  organized  into  550 
occupational  groups.  The  DOT  was  revised  several  times  over  the  years,  with  the  last  revision 
occurring  in  1991  (Dye  &  Silver,  1999). 

While  several  data  gathering  methods  were  tried  over  the  years,  the  philosophical 
underpinnings  and  methodological  basis  of  the  1991  edition  of  the  DOT  still  reflected  “the  work 
structure  of  mid-20th  century  America”  (Dye  &  Silver,  1999,  p.  11).  Additionally,  the  time  and 
expense  involved  in  updating  it  had  become  prohibitive,  and  there  were  some  concerns  about  the 
comprehensiveness  of  its  descriptions.  For  these  reasons,  DOL  formed  the  Advisory  Panel  on  the 
Dictionary  of  Occupational  Titles  (APDOT),  which  recommended  development  of  an 
occupational  database  and  outlined  a  content  model  for  it.  First,  a  prototype  0*NET  database 
was  developed  on  a  sample  of  occupations  (Peterson  et  al.,  1991).  Then,  a  series  of  efforts  began 
to  populate  the  database  with  descriptions  of  all  812  occupations  in  the  Standard  Occupational 
Classification  (SOC,  http://www.bls.gov/soc/).  To  ensure  a  controlled  data  collection  and 
management  process,  occupational  data  have  been  collected  in  groups  or  “analysis  cycles.”2 
Approximately  1 00  occupations  have  been  included  in  each  cycle.  Cycle  9,  which  is  currently 
underway,  will  complete  the  initial  population  of  0*NET.  Additional  information  about  Q*NET 


2  For  a  description  of  the  0*NET  Data  Collection  Publication  Schedule,  see  www.onetcenter.org. 
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is  available  in  the  paper  by  Tsacoumis  (2007),  which  appears  in  Appendix  A,  or  at  the  0*NET 
website  ('http://online.onetcenter.org/). 

It  is  worth  pointing  out  that  neither  the  DOT  nor  the  0*NET  included  military 
occupations.  However,  some  of  the  descriptor  sets  in  0*NET  are  strongly  tied  to  military 
research,  particularly  the  0*NET  abilities.  Fleishman,  Costanza,  and  Marshall-Mies  (1999)  cite  a 
number  of  military  studies  conducted  in  the  development  and  validation  of  the  Abilities 
Requirements  Taxonomy  (Fleishman  &  Quaintance,  1984),  which  provided  the  basis  for  the 
0*NET  abilities. 

The  Office  of  the  Secretary  of  Defense  (OSD)  maintains  a  crosswalk  between  0*NET 
and  military  occupations.  The  Services  provide  information  to  OSD  about  military  occupational 
codes.  OSD  assigns  the  Service  occupation  code  (e.g.,  MOS)  to  a  higher  level  occupational  code 
and  cross  walks  that  code  to  the  SOC.  The  Services  provide  information  to  OSD  in  various 
forms;  there  is  not  a  standardized  format  for  providing  information  (personal  communication, 
Steve  Reardon,  25  July  2007).  Consequently,  the  crosswalk  is  based  on  broad  occupational 
identifiers,  not  on  occupation  analysis  data  for  occupational  descriptors. 

Arm y  Occupa/iona /  A na/ysis 

The  Army  has  a  long  history  of  conducting  specific  occupational  analyses  to  serve  the 
needs  of  training,  selection,  and  classification.  In  the  1960s,  the  U.S.  Total  Army  Personnel 
Command  (PERSCOM)  (currently  the  Human  Resources  Command)  established  the 
Occupational  Analysis  Program  to  assist  in  resolving  Manpower,  Personnel  and  Training  (MPT) 
issues  associated  with  the  Army’s  role  in  Vietnam  (Brady,  2004).  Over  the  course  of  the  war,  the 
number  of  Military  Occupational  Specialties  (MOS)  had  increased  dramatically,  and  the  Army 
needed  occupation  analysis  data  to  assist  in  creating/merging  MOS  and  identifying  training 
requirements  for  them.  In  1 972,  the  Army  adopted  the  Comprehensive  Occupational  Data 
Analysis  Programs  (CODAP)  approach,  pioneered  by  the  Air  Force  Human  Resource  Laboratory 
in  the  late  1 950s.  CODAP  is  a  collection  of  programs  for  processing  task  inventory  data;  it  yields 
a  series  of  standardized  reports. 

The  next  major  change  was  in  1994  when  the  occupational  analysis  function  moved  from 
PERSCOM  to  ARI’s  new  Occupational  Analysis  Office  (OAO),  where  it  resides  today  (Brady, 
2004).  ARI  created  the  Occupational  Data  Analysis,  Requirements,  and  Structure  (ODARS) 
program  to  reduce  turnaround  time  and  flexibility  in  occupational  analyses  efforts.  In  this  system 
ARI  worked  with  the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  schools  to 
determine  what  tasks  were  performed;  developed  and  administered  a  task  survey;  and  organized, 
analyzed,  and  reported  the  data.  The  most  recent  and  substantive  change  in  ARI’s  OAO  efforts 
has  been  a  move  to  design  an  occupational  analysis  system  that  enhances  the  self-sufficiency  of 
TRADOC  schools  and  centers  (Brady,  2004).  ARI  created  a  software  package,  the  Automated 
Survey  Generator  (AUTOGEN),  to  develop,  gather,  analyze,  and  report  occupational  analysis 
information  and  implemented  AUTOGEN  in  schools  and  centers.  In  this  new  model,  ARI’s 
OAO  staff  serves  as  expert  advisors  to  the  schools  and  resources  for  special  projects. 

Additionally,  ARI  conducts  occupation  analyses  for  selection  and  classification  purposes  on  a 
project-by-project  basis. 
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In  sum,  the  Army’s  current  approach  to  job  analysis  is  designed  to  meet  training  needs. 
Recently,  the  Army  has  desired  improvements  to  selection  and  classification  research,  and  to 
achieve  those  improvements  new  job  analysis  tools  are  needed  (Campbell  et  al.,  2006).  Army 
recruits  are  classified  into  MOS  based  on  their  scores  on  the  Armed  Services  Vocational 
Aptitude  Battery  (ASVAB)3;  those  scores  are  input  into  equations  that  estimate  how  well  recruits 
are  likely  to  perform  in  the  MOS.  These  classification  equations  are  derived  from  validity  studies 
in  which  scores  on  predictors  like  the  ASVAB  are  compared  to  job  performance  criteria.  Having 
a  meaningful  and  reliable  criterion  measure  is  a  critical  component  for  ensuring  the  success  of 
classification  research.  Yet  collecting  criterion  data  for  a  sufficient  number  of  jobs  to  meet  the 
Army’s  classification  research  needs  is  laborious  and  expensive. 

In  2006,  the  Army  convened  a  panel  of  experts  to  make  recommendations  for  how  the 
Army  should  obtain  criterion  data  in  an  ongoing  fashion  and  ensure  the  transportability  of 
validities  within  job  families  (Campbell  et  al.,  2006).  The  panel  concluded  that  a  key  component 
of  the  solution  was  to  have  a  solid  job  analysis  system.  The  panel  articulated  the  design 
characteristics  that  would  be  highly  desirable  for  an  Army-specific  system,  and  many  of  those 
features  are  key  elements  of  0*NET.  For  example,  the  panel  recommended  that  the  Army  job 
analysis  system  use  a  common  language  for  describing  similarities  and  differences  in  MOS  and 
that  it  consist  of  cross-MOS  descriptors  as  well  as  occupation-specific  ones.  Therefore,  the 
evaluation  of  0*NET  for  describing  Army  jobs  is  an  important  step  in  the  development  of  a  job 
analysis  system  that  can  serve  human  resource  functions  other  than  training. 

Overview  of  Approach 

The  present  evaluation  focused  primarily  on  the  usefulness  of  the  0*NET  system  for 
Army  occupational  analysis  for  selection  and  classification  purposes.  That  is,  any  occupational 
analysis  system  has  utility  only  if  it  serves  the  purposes  it  is  intended  to  serve  (Society  for 
Industrial  and  Organizational  Psychology,  Inc.,  2003).  The  0*NET,  which  is  organized  around  a 
content  model,  contains  a  wealth  of  data,  and  evaluating  all  of  0*NET  for  all  possible  human 
resource  needs  would  be  beyond  the  scope  of  this  effort.  The  0*NET  content  model  is 
comprised  of  six  domains,  including:  (a)  worker  characteristics  (e.g.,  abilities,  interests,  work 
styles),  (b)  worker  requirements  (e.g.,  skills,  knowledge,  and  education),  (c)  experience 
requirements  (e.g.,  training),  (d)  occupation  requirements  (e.g.,  generalized  work  activities,  work 
context),  (e)  workforce  characteristics,  and  (f)  occupation-specific  information  (e.g.,  tasks,  tools, 
and  technology).  Within  each  content  model  domain,  information  is  organized  by  different  levels 
of  description.  We  focused  on  the  appropriateness  of  0*NET  descriptors  that  would  typically  be 
used  in  an  Army  occupation  analysis  for  selection  and  classification  purposes.  These  included 
abilities,  skills,  generalized  work  activities  [GW As],  and  work  context. 

Our  approach  was  designed  to  address  the  following  six  questions: 

1 .  Are  Army  occupations  rated  reliably  using  0*NET  rating  scales? 


3  Other  factors  such  as  the  availability  of  training  seats  also  influence  classification  assignments. 
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2.  Do  ratings  on  0*NET  rating  scales  differentiate  Army  occupations? 

3.  How  well  do  subject  matter  expert  (SME)  and  analyst  ratings  agree? 

4.  Are  ratings  on  Army  occupations  similar  in  quality  to  ratings  on  civilian  occupations? 

5.  Are  Army  occupational  profiles  similar  to  those  for  their  civilian  counterparts? 

6.  How  well  do  0*NET’s  work  requirements  descriptors,  particularly  GW  As,  cover 

Army  job  requirements? 

Addressing  the  first  five  questions  required  collection  of  ratings  on  the  selected  0*NET 
descriptors — abilities,  skills,  GW  As,  and  work  context — for  target  Army  occupations.  Our  plan 
was  to  produce  data  for  the  military  occupations  that  could  be  compared  to  civilian  0*NET  data. 
Therefore,  it  was  important  to  follow  processes  currently  used  by  0*NET  for  data  collection.  For 
0*NET,  job  incumbents  provide  ratings  on  occupational  tasks,  skills,  GWAs,  and  work  context 
areas.  Ability  ratings  for  occupations  are  collected  from  trained  analysts.4  Analysts  have  also 
been  trained  to  rate  skills,  although  until  recently  those  data  have  been  used  only  experimentally. 
Accordingly,  we  planned  to  collect  skill,  GWA,  and  work  context  data  from  SMEs  for  each 
target  Army  occupation.  We  planned  to  collect  analyst  ratings  on  the  abilities  and  skills  for  target 
Army  occupations,  following  the  procedures  used  in  collecting  0*NET  analyst  ratings  for  SOC 
occupations  (Donsbach,  Tsacoumis,  Sager,  &  Updegraff,  2003).  In  the  0*NET  analyst  training, 
analysts  rate  occupations  using  a  standard  set  of  descriptive  materials — tasks  for  the  occupation 
and  incumbents’  ratings  of  those  tasks,  GWAs,  and  work  context  descriptors.  To  use  a  parallel 
approach  in  the  evaluation,  we  needed  to  develop  task  lists  for  the  target  Army  occupations.  In 
sum,  the  plan  required  collecting  task,  skill,  GWA,  and  work  context  ratings  from  SMEs  and 
ability  ratings  from  analysts.  To  allow  comparison  of  SME  and  analyst  data,  we  also  asked 
SMEs  to  rate  abilities  and  analysts  to  rate  skills. 

The  sixth  question  required  somewhat  different  data.  If  0*NET  GWAs  appeared  to  be 
related  to  or  “cover”  all  or  many  of  the  Army  job  requirements  that  are  common  across 
occupations,  here  called  Major  Duties  (MDs),  then  they  would  potentially  be  quite  useful  for 
providing  a  higher-order  description  of  Army  occupations  (i.e.,  one  that  would  encompass  the 
major  duties  performed  in  the  Army).  If  not,  additions  or  revisions  to  the  GWAs  might  be 
needed.  In  preparation  for  this  assessment  of  coverage,  we  developed  a  major  duties  list  based  on 
prior  research  and  obtained  feedback  on  it  from  Army  SMEs.  To  make  this  evaluation,  we  asked 
non-incumbent  raters  to  rate  the  degree  to  which  each  MD  was  “covered”  by  each  GWA,  with  no 
occupation  specifically  identified  (i.e.,  the  MD  might  be  imagined  to  be  a  part  of  any  of  several 
occupations;  no  particular  occupation  was  implicated). 


Overview  of  Report 

The  remaining  chapters  of  this  report  describe  the  procedures  and  results  of  the 
evaluation.  In  Chapter  II,  we  describe  the  SMEs  and  analysts  and  the  data  collection  efforts. 
Chapter  III  addresses  questions  1-5  listed  earlier  through  comparisons  of  intraclass  correlation 
coefficients,  convergent/discriminant  analyses,  and  descriptive  statistics.  Chapter  IV  describes 


4  The  rationale  here  is  that  job  incumbents  are  more  likely  to  be  familiar  with  the  day-tp-day  duties  and  conditions  of 
their  jobs,  while  trained  analysts  are  likely  to  be  able  to  understand  the  ability  constructs  and  make  ability  ratings 
(Tsacoumis,  2007). 
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the  procedures  and  results  of  the  evaluation  of  the  GWAs  against  MDs,  and  Chapter  V  provides 
a  discussion  of  the  evaluation  results. 
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II.  Method  of  Describing  Army  Occupations  Using  0*NET 

One  key  component  of  the  evaluation  was  to  examine  how  effectively  different  types  of 
raters  could  use  the  0*NET  descriptors  to  describe  Army  occupations.  We  included  three 
different  types  of  raters — Subject  Matter  Experts  (SMEs),  project  analysts,  and  0*NET  analysts. 

•  SMEs  were  NCOs  or  officers  with  several  years  of  experience  in  the  Army  and  their 
occupations,  and  who  thus  had  first-hand  observation  and  experience  of  the  selected 
target  occupations  (described  later  in  this  chapter).  Prior  to  SME  workshops,  we  drafted 
lists  of  the  major  duties  and  key  work  activities  (KWAs)  for  their  occupation.  In  the 
workshops,  SMEs  reviewed  and  finalized  the  occupation-specific  materials  and  then 
rated  0*NET  abilities,  skills,  GWAs,  and  work  context  variables. 

•  0*NET  trainers  allowed  us  to  collect  our  analyst  data  within  the  context  of  scheduled 
training  and  data  collection  for  0*NET.  0*NET  and  project  analysts  attended  a  2  Vi  day 
training  session  on  rating  abilities  and  skills.  KWAs,  GWAs,  and  work  context  ratings 
from  the  SME  workshops  served  as  stimulus  materials  for  analysts  to  use  in  making  their 
ratings. 

o  0*NET  analysts  are  required  to  have  at  least  two  years  of  work  experience,  have 
completed  two  years  of  graduate  education  in  a  human  resources  related  field,  and 
have  completed  courses  in  job  analysis  and  research  methods.  Of  the  1 6  analysts 
who  participated,  all  but  one  had  participated  in  previous  data  collections  for 
0*NET. 

o  Project  analysts  were  generally  graduate-level  trained  psychologists  with  varying 
levels  of  experience  in  researching  Army  occupations.  Seven  of  the  eight  analysts 
were  HumRRO  employees;  one  was  a  highly  experienced  occupational  analyst 
from  ARI. 

•  We  sought  eight  raters  of  each  type  because  0*NET  researchers  imposed  a  rule  of  eight 
analyst  ability  ratings  for  each  occupation  based  on  prior  reliability  studies.5  For 
interpretation  of  data  in  Chapter  III,  it  is  important  to  note  that  SMEs  made  ratings  only 
for  their  target  occupation,  and,  in  the  end,  their  numbers  varied.  The  eight  project 
analysts  rated  all  target  occupations.  While  1 6  0*NET  analysts  participated  in  the 
research,  each  analyst  only  rated  half  of  the  target  occupations.  Therefore,  there  were 
eight  0*NET  analyst  ratings  for  each  occupation. 

Given  these  backgrounds  and  procedures,  the  SMEs  were  likely  to  be  more  familiar  with 
the  target  occupations,  in  view  of  their  first-hand  experience  in  them,  but  likely  to  have  less 
context  from  which  to  make  their  ratings  in  terms  of  experience  in  other  occupations  or  with  the 
use  of  occupational  analysis  rating  scales.  On  the  other  hand,  the  0*NET  analysts  have 
experience  using  standardized  occupational  information  and  the  greatest  familiarity  with  the 
specific  occupational  analysis  scales  used  here,  but  the  least  first-hand  knowledge  and 


5  Prior  0*NET  research  suggested  that  eight  raters  would  provide  sufficient  reliability  for  ability  ratings  (Donsbach 
et  al.,  2003). 
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experience  with  the  focal  occupations.  The  project  analysts  fall  somewhere  in  between  these  two 
extremes. 

This  chapter  describes  the  methods  used  to  collect  SME,  project  analyst,  and  0*NET 
analyst  data6. 


SME  Data  Collection  Methods 

/den  if  cation  of Targe i  Occupations 

Identifying  target  occupations  required  balancing  a  number  of  competing  concerns.  We 
wanted  jobs  that  were  populous  and  had  extant  job  analysis  information  that  we  could  use  as  a 
starting  point.  We  also  wanted  some  jobs  that  were  Army-specific  and  others  that  would  have 
civilian  counterparts.  We  needed  to  include  both  officer  and  enlisted  jobs.  Finally,  this  project 
was  conducted  during  wartime,  making  accessibility  of  SMEs  very  difficult. 

We  made  a  list  of  a  dozen  or  so  occupations  for  possible  study.  Then,  ARI  worked  with 
training  schools  for  those  occupations  to  determine  accessibility  of  SMEs.  In  the  end,  four 
enlisted  and  four  officer  occupations  were  targeted.  The  officer  occupations  were  counterparts  to 
the  enlisted  ones.  They  were: 

3 1 A  Military  Police  (Officer)  3 1 B  Military  Police  (Enlisted) 

88A  Transportation  (Officer)  88M  Motor  Transport  Operator  (Enlisted) 

1 9 A  Armor  (Officer)  1 9K  M 1  Armor  Crewman  (Enlisted) 

25A  Signal  (Officer)  25U  Signal  Support  Systems  Specialist  (Enlisted) 

Note  that  officer  jobs  are  designated  by  a  suffix  of  “A”.  For  selection  and  classification 
purposes,  ARI  typically  needs  job  information  on  entry-level  jobs  (Skill  Level  1  [SL1]  enlisted; 
second  lieutenant  [2LT]  officers).  Therefore,  our  focus  for  the  data  collection  was  on  entry-level 
job  requirements. 

Deve/opment  of  Key  iVork  Activities  and  May  or  Duties 

With  help  from  ARI,  we  obtained  as  much  information  as  possible  about  each 
occupation.  Information  sources  included: 

•  Job  analyses  from  prior  ARI  projects 

•  Job  analyses  conducted  for  training  purposes  through  ARI’s  Occupational  Analysis  group 

•  Soldier’s  Training  Manuals  (STMs)  and  Field  Manuals  (FMs) 

•  Mission  Training  Plans  (MTPs) 

•  Programs  of  Instruction  (POIs) 

•  Officer  Foundation  Standards  (OFS) 

•  Occupational  information  from  similar  civilian  occupations  (e.g.,  from  0*NET) 


6  It  is  also  important  to  note  that  data  collected  for  this  project  also  served  the  needs  of  a  complementary  job  analysis 
pilot  study  underway  at  ARI. 
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Not  all  sources  were  available  for  all  occupations.  For  example,  the  OFSs  were  quite 
useful,  but  they  are  not  available  for  all  officer  occupations. 

We  identified  two  types  of  occupational  descriptors  for  development.  The  first  type  we 
termed  key  work  activities  (KWAs).  These  are  less  specific  descriptions  of  work  behavior  than 
are  job  tasks.  A  KWA  was  defined  as  a  statement  describing  a  series  or  clustering  of  related 
behaviors  a  Soldier  (or  Officer)  performs  to  achieve  a  specific  work  objective.  KWAs  were 
constructed  to  be  as  behaviorally  homogeneous  as  possible.  For  example,  “maintain  individual 
weapon”  and  “aim  and  fire  individual  weapon”  would  be  useful  KWAs  because  they  describe 
different  types  of  behaviors.  “Maintain  and  fire  individual  weapon”  would  not  meet  our  criterion 
of  behavioral  homogeneity.  We  expected  the  KWAs  to  be  occupation-specific,  although  some  of 
them  might  be  relevant  to  several  occupations. 

In  contrast  to  KWAs,  the  second  type  of  descriptor,  major  duties  (MDs),  consisted  of 
broader  statements  intended  to  apply  across  occupations.  A  major  duty  was  defined  as  a 
statement  describing  a  definable  and  nontrivial  duty,  or  responsibility,  which  a  Soldier  (or 
Officer)  is  accountable  to  perform,  which  has  stakeholders  to  whom  the  results  (i.e.,  outputs)  are 
important  and  meaningful,  and  which  entails  work  of  significant  complexity  and  duration. 

“Repair  mechanical  systems”  and  “drive  wheeled  vehicle”  are  examples  of  major  duties.  Specific 
guidelines  for  writing  KWAs  and  MDs  appear  in  Ingerick  (2007). 

We  assigned  staff  to  work  with  particular  occupations  and  conduct  training  on  writing  the 
KWA  statements.  Staff  members  were  told  to  first  review  all  the  source  materials  for  their 
assigned  occupation  and  to  put  the  source  materials  in  spreadsheets.  Source  materials  were  coded 
and  KWAs  based  on  the  sources  retained  the  source  code.  Staff  members  were  told  to  sort 
individual  tasks  into  categories,  based  on  behavioral  homogeneity,  and  to  write  a  KWA 
statement  to  cover  the  behaviors  in  those  tasks. 

Staff  assigned  to  write  MDs  were  told  to  begin  by  reviewing  existing  taxonomies,  in 
particular:  (a)  two  Synthetic  Validity  (SYNVAL)  project  taxonomies  (task  categories  and  job 
activities;  Peterson,  Owens-Kurtz,  Hoffman,  Arabian,  &  Whetzel,  1990)  (b)  Army-Wide 
performance  requirements  from  Army21  projects  (Knapp  &  Campbell,  2006;  Sager,  Russell, 
Campbell,  &  Ford,  2005),  (c)  performance  dimensions  from  military  performance  models,  and 
(d)  task  categories  from  manuals  and  job  analyses.  Staff  were  asked  to  copy  sources  into  a 
spreadsheet,  content  analyze  the  content,  and  sort  like  statements  together.  This  process 
continued  until  staff  members  were  comfortable  that  they  had  identified  a  reasonable  set  of  MDs 
(the  final  total  was  98)  that  covered  statements  in  other  taxonomies. 

SM£  Workshops 

Once  KWAs  and  MDs  were  drafted,  the  next  step  was  to  plan  SME  workshops  and  train 
staff  in  workshop  procedures.  The  general  structure  for  the  workshops  was  to  have  two  4-hour 
blocks  as  shown  in  Figures  1  and  2. 
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1 .  In-Processing  ( 1 0  min.) 

2.  Review  Major  Duties  (30  min.) 

-  Introduce  and  define  MDs. 

-  Identify  MDs  relevant  to  Skill  Level  1  (or  entry  officer)  for  this  MOS 

3.  Review  MOS-Specific  and  Army-Wide  Job  Descriptions  (30  min.) 

What  is  missing? 

-  What  is  irrelevant  or  misstated? 

4.  Collect  KWA  ratings  of  Frequency,  Importance,  and  Performance  Variability  (25  min.) 

5.  Break  (15  min.)  (HumRRO  begins  analysis  of  KWA  ratings) 

6.  Collect  work  context  ratings  on  two  work  context  questionnaires  (JA  Pilot  and  0*NET)  (40  min.) 

7.  Break  (15  min.)  (HumRRO  analyzes  KWA  and  work  context  data) 

8.  Review  and  Discuss  KWA  ratings  (45  min.) 

-  Present  means  and  SDs  of  frequency  ratings  (ordered  from  high  to  low) 

o  Do  the  means  make  sense  (high  frequency  compared  to  low  frequency)? 
o  Discuss  KWAs  with  high  SDs.  Does  the  high  SD  reflect  the  reality  of  the  job  or  is  it  due  to  a 
problem  with  KWA  wording? 

-  Present  means  and  SDs  of  importance  ratings  (ordered  from  high  to  low) 

o  Do  the  means  make  sense  (highest  to  lowest)? 
o  Discuss  KWAs  with  high  SDs. 

9.  Review  and  Discuss  work  context  ratings  (30  min.) 

-  Present  means  and  SDs  of  ratings 

o  Do  the  means  make  sense  (high  frequency  compared  to  low  frequency)? 

_ o  Discuss  work  context  items  with  high  SDs. _ 

Figure  1.  Block  1  agenda. 


During  the  first  block,  we  collected  data  on  occupation-specific  information:  MDs, 
KWAs,  and  0*NET  work  context  variables.  In  the  second  block,  we  administered  0*NET 
abilities,  skills  and  GWA  questionnaires.  After  each  questionnaire,  we  analyzed  the  data  and 
presented  it  back  to  SMEs  for  comment. 

Ideally,  we  planned  for  Blocks  1  and  2  to  be  held  on  different  days  to  allow  our  staff  time 
to  analyze  and  synthesize  the  data.  In  reality,  we  had  to  be  flexible  to  accommodate  the  Army’s 
needs.  The  31 A  and  31 B  workshops  were  conducted  as  planned.  For  both  88 A  and  88M  we 
broke  the  agenda  into  four  2-hour  workshops  held  over  four  days.  For  19A,  19K,  25 A,  and  25U 
we  conducted  the  two  blocks  back-to-back  in  full-day  workshops.  The  actual  time  spent  on 
elements  in  the  agendas  also  varied.  When  we  ran  short  of  time,  we  reduced  the  time  set  aside  for 
discussion  of  the  ratings. 
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1 .  In-Processing  ( 1 0  min.) 

2.  Review  and  Discuss  MOS-Specific  and  Army  Wide  Job  Description  Results  from  Block  1  Session  (45 
min.) 

3.  Introduce  0*NET  (10  min.) 

-  What  is  it,  in  general  and  what  it  is  used  for? 

-  Provide  overview  of  components  we  will  focus  on  (i.e.,  GW  As,  Abilities,  Skills) 

4.  Collect  Ability  and  Skill  Ratings  (60  min.) 

Brief  training 

-  Make  importance  and  level  ratings  (HumRRO  collects  abilities  data  while  SMEs  rate  skills  and  begins 
analysis) 

5.  Break  (10  min.) 

6.  Review  and  Discuss  Ability  and  Skill  ratings  (Make  sense?  Reasons  for  high  SDs)  (30  min.) 

-  Abilities 

o  Present  means  and  SDs  of  importance  ratings 
o  Present  means  and  SDs  of  level  ratings 

-  Skills 

o  Present  means  and  SDs  of  importance  ratings 
o  Present  means  and  SDs  of  level  ratings 

7.  Collect  GWA  Ratings  (30  min.) 

-  Brief  training 

Make  importance  and  level  ratings 

8.  Break  (15  min.) 

9.  Review  and  Discuss  GWA  ratings  (Make  sense?  Reasons  for  high  SDs)  (30  min.) 

-  Present  means  and  SDs  of  importance  ratings 

_  Present  means  and  SDs  of  level  ratings  _ 

Figure  2.  Block  2  agenda. 


Major  Duties.  The  SMEs  first  reviewed  the  list  of  major  duties  generated  by  the  research 
team  prior  to  the  workshops.  The  purpose  of  this  activity  was  to  determine  which  major  duties 
were  relevant  to  entry  level  in  the  MOS  and  to  generate  more  specific  work  activities  relevant  to 
the  major  duties.  We  asked  SMEs  to  review  the  entire  list  of  major  duties  and  nominate  ones  that 
were  most  relevant  to  entry  level  in  the  occupation.  Because  so  many  MDs  are  at  least  somewhat 
relevant,  we  also  asked  the  SMEs  to  identify  the  MDs  that  best  distinguish  the  occupation  from 
others.  We  kept  notes  on  the  discussions  and,  after  completion  of  all  the  workshops,  we  made 
revisions  to  the  list  of  MDs. 

Key  Work  Activities.  We  presented  draft  lists  of  Army-wide  and  occupation-specific 
KWAs  to  SMEs  in  the  workshop.  We  led  a  discussion  of  the  KWAs,  making  edits  in  an  Excel 
spreadsheet,  projected  so  that  SMEs  could  see  and  approve  the  edits.  When  SMEs  were  satisfied 
with  the  KWA  list,  we  printed  rating  forms.  SMEs  rated  the  final  KWAs  in  terms  of  importance, 
frequency,  and  performance  variability.  We  needed  importance  ratings  for  use  in  the  analyst  data 
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collection.  The  frequency  and  performance  variability  rating  scales  served  the  needs  of  another 
job  analysis  project.  The  three  rating  scales  are  show  in  Figure  3. 


How  important  is  this  KWA  for  entry-level  performance  in  this  job? 

1  =  Not  Important 

2  -  Somewhat  Important 

3  =  Important 

4  =  Very  Important 

5  =  Extremely  Important 

How  frequently  do  entry-level  enlisted  Soldiers  in  this  job  perform  this  KWA? 

1  -  Once  in  1  to  2  years 
2-2-4  times  a  year 

3  -  Once  or  twice  a  month 

4  -  Once  or  twice  a  week 

5  -  Once  or  twice  a  day 

6  -  Once  an  hour 

7  -  Several  times  an  hour 

Provided  sufficient  equipment  and  resources  are  available,  what  percentage  of  entry-level  enlisted  Soldiers 
in  this  job  perform  this  KWA  to  standard? 

1  -  0%-20% 

2  -  21%-40% 

3-41  %-60% 

4  -  61%-80% 

5  -  8 1  %- 1 00% 


Figure  3.  KWA  rating  scales. 

0*NET  Rating  Scales.  For  the  0*NET  work  context,  ability,  skill,  and  GWA  ratings,  we 
used  0*NET  rating  scales  downloaded  from  0*NET  Online.  However,  we  did  make  one  minor 
modification  to  the  stem  questions  on  all  the  questionnaires.  0*NET  questionnaires  ask 
incumbents  to  rate  your  current  job.  Since  we  were  asking  higher  level  NCOs  and  officers  to 
"make  ratings  for  entry-level  occupations,  we  altered  the  stem  to  refer  to  the  entry-level  job. 

Work  Context  The  0*NET  work  context  questionnaire  (Strong  et  al.,  1999)  contains  57 
items  that  ask  raters  about  interpersonal  relationships,  physical  work  conditions,  and  structural 
job  characteristics.  The  rating  scales  vary  across  items.  Some  of  the  scales  are  importance 
scales;  others  are  frequency  scales  or  scales  customized  to  the  question.  An  example  item 
appears  in  Figure  4.  SMEs  made  ratings  on  the  57  0*NET  items. 

0*NET  Abilities  and  Skills.  SMEs  used  the  0*NET  ability  (Fleishman  et  al.,  1999)  and 
skill  (Mumford,  Peterson,  &  Childs,  1999)  questionnaires  to  describe  the  abilities  and  skills 
required  by  Skill  Level  1  for  their  occupation.  SMEs  rated  each  of  the  52  0*NET  abilities  and  35 
0*NET  skills  on  a  5-point  importance  scale  and  a  7-point  level  scale  as  illustrated  in  Figure  5. 
We  provided  only  a  very  brief  orientation  to  the  scales,  stressing  the  importance  of  reading  the 
anchors. 
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1 .  How  often  does  this  entry-level  job  in  your  MOS/Branch  require  face-to-face 
discussions  with  individuals  and  within  teams? 


Once  a  year  or  more  Once  a  month  or  more  Once  a  week  or  more 
Never  but  not  every  month  but  not  every  week  but  not  every  day  Every  day 

(D - @ - ®- - (D - ® 


Figure  4.  Example  work  context  item. 


A.  How  important  is  ORAL  COMPREHENSION  to  the  performance  of  entry-level 
Soldiers  in  your  MOS/Branch  ? 


Not  Somewhat  Very  Extremely 

Important*  Important  Important  Important  Important 

(D - (D - ® - @ - ® 

*  If  you  marked  Not  Important,  skip  LEVEL  below  and  go  on  to  the  next  activity. 

B.  What  level  of  ORAL  COMPREHENSION  is  needed  by  entry-level  Soldiers  to  perform  in 
your  MOS/Branch ? 

Understand  a  Understand  a  coach’s  oral  Understand  a  lecture 

television  commercial  instructions  for  a  sport  on  advanced  physics 

I  1  1 

(D - ® - CD - (I) - (D - © - © 

Highest  Level 


Figure  5.  Example  0*NET  importance  and  level  rating  scales. 

0*NET  Generalized  Work  Activities.  SMEs  rated  the  0*NET’s  41  GW  As  using  the 
0*NET  GWA  questionnaires.  The  questionnaires  call  for  ratings  on  two  scales,  importance  and 
level,  formatted  like  those  shown  in  Figure  5  for  abilities  and  skills. 

General  process.  After  SMEs  made  ratings  on  descriptors,  we  entered  and  analyzed  data, 
and  presented  the  data  back  to  the  SMEs.  We  discussed  the  rank  ordering  of  the  descriptors, 
paying  particular  attention  to  descriptors  that  had  high  standard  deviations,  which  could  reflect 
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(a)  true  differences  in  the  occupation  across  deployments/settings  or  (b)  a  misunderstanding  of 
the  item  or  scale.  We  took  notes  on  SME  comments. 

SMEs.  Most  of  the  enlisted  participants  were  NCOs  who  were  currently  instructors  for 
the  MOS;  88M  participants  were  attending  the  Advanced  Non-Commissioned  Officer  Course 
(ANCOC).  Officers  were  typically  individuals  who  were  attending  a  training  course  or 
instructors  for  training  courses.  As  shown  in  Table  1,  we  were  unable  to  obtain  eight  SMEs  for 
each  occupation.  88  A,  19A,  and  19K  had  fewer  than  8  participants.  Also,  two  of  the  five  88  A 
SMEs  had  to  leave  the  workshop  before  completing  the  ability,  skill,  and  GWA  ratings.  Most  of 
the  participants  had  recently  been  deployed  in  Iraq.  Many  of  them  had  been  on  numerous 
deployments  in  many  locations.  Some  of  the  officers  had  been  enlisted  personnel  prior  to 
becoming  officers. 


Table  1.  SME  Workshop  Participants  by  Level  and  Occupation 


Level 

31A 

31B 

88A 

88M 

19A 

19K 

25A 

25U 

E-5,  SGT 

1 

3 

E-6,  SSG 

3 

1 

5 

2 

E-7,  SFC 

5 

8 

1 

4 

0-2,  1LT 

I 

1 

0-3,  CPT 

8 

4 

1 

9 

0-4,  MAJ 

1 

3 

Total 

9 

9 

5 

9 

4 

6 

10 

9 

Results.  Table  2  shows  descriptive  statistics  for  the  KWA  ratings.  There  was  range 
restriction  in  the  KWAs  in  terms  of  importance.  That  is,  virtually  all  of  the  KWAs  received  mean 
ratings  above  “3”  on  the  5-point  rating  scale.  This  is  probably  because  our  method  of  KWA 
generation  eliminated  less  important  activities;  it  is  also  possible  that  SMEs  were  simply  inclined 
to  give  high  ratings  of  importance.  There  was  more  variability  in  the  frequency  and  performance 
variability  ratings.  A  separate  research  note  (Russell,  Sinclair,  Erdheim,  Ingerick,  Owens, 
Peterson,  &  Pearlman,  2008)  provides  tables  of  descriptive  statistics  for  KWA  ratings  for  each 
occupation. 


Table  2.  Descriptive  Statistics  for  Key  Work  Activity  Ratings 


Occupation 

n 

Number  . 
of 

KWAs 

Importance  (1  to  5) 

Frequency  (1  to  7) 

Performance 
Variability  (1  to  5) 

Mean 

of  A/s 

Mdn 

ofSDs 

Mean 

of  A/s 

Mdn 

ofSDs 

Mean 

of  A/s 

Mdn 

ofSDs 

31 A 

9 

45 

4.23 

.73 

3.44 

1.39 

3.42 

.88 

31B 

9 

38 

4.32 

.83 

3.63 

1.41 

3.36 

1.20 

88A 

5 

28 

4.26 

.63 

4.00 

.55 

4.38 

.58 

88M 

9 

23 

4.65 

.71 

4.41 

1.51 

3.70 

1.26 

19A 

4 

46 

4.16 

.50 

4.02 

.82 

3.90 

.58 

19K 

6 

37 

4.06 

.82 

3.52 

1.15 

2.87 

1.11 

25A 

10 

53 

4.56 

.71 

4.08 

1.34 

4.23 

1.32 

25U 

9 

35 

4.40 

.67 

3.86 

1.22 

3.62 

1.00 

Total  M 

38.13 

4.33 

.70 

3.87 

1.17 

3.69 

.99 

Officer  M 

43.00 

4.30 

.64 

3.89 

1.03 

3.98 

.84 

Enlisted  M 

33.25 

4.36 

.76 

3.86 

1.32 

3.39 

1.14 

14 


To  examine  the  reliability  of  the  KWA  ratings,  we  calculated  the  intraclass  correlations 
(ICC  [3,  it];  Shrout  &  Fleiss,  1979)  among  the  SMEs’  ratings  to  look  at  consistency  across 
KWAs  within  occupations.7  As  shown  in  Table  3,  there  was  a  great  deal  of  variability  in  ICCs 
for  different  occupations.8  Reliabilities  for  the  two  officer  occupations,  31 A  and  25A  were 
particularly  low.  Using  Nunnally’s  (1967,  p.  225)  equation,  we  estimated  that  the  following 
numbers  of  raters  would  have  been  needed  to  obtain  a  desired  multi-rater  ICC  of  .80  (McCloy, 
Waugh,  &  Medsker,  1998):  129,  21,  12,  17,  10,  8,46,  and  13  for  31  A,  3 IB,  88A,  88M,  19A, 

1 9K,  25A,  and  25U,  respectively.9 

One  reason  for  lower  ICCs  may  have  been  differential  emphases  by  SMEs  on  stateside, 
in-garrison  activities  vs.  deployment  activities.  For  example,  31A  and  3 IB  occupations  in 
garrison  involve  police  work  similar  to  that  of  civilian  police  (e.g.,  responding  to  domestic 
disputes),  but  in  Iraq  31 A  and  31 B  have  responsibility  for  guarding  prisoners/detainees  and 
going  on  patrol.  In  the  31 A  workshop,  for  example,  some  officers  were  active  duty  and  others 
were  in  the  National  Guard,  and  some  had  been  in  the  enlisted  ranks  prior  to  becoming  officers. 
They  had  diverse  experiences.  The  garrison/deployment  issue  came  up  in  every  workshop  and 
was  an  issue  for  all  jobs,  but  it  may  have  affected  some  jobs  more  than  others.  To  try  to 
minimize  the  issue,  we  asked  SMEs  to  respond  from  the  perspective  of  their  recent  deployments. 
Regardless  of  our  instructions,  their  perceptions  continued  to  influence  their  ratings.  In 
discussions  of  ratings,  garrison  vs.  deployment  was  an  issue,  particularly  for  the  frequency  and 
performance  variability  rating  scales. 

Another  reason  for  lower  ICCs  might  have  to  do  with  the  variability  of  officer 
occupations  and  experiences.  For  example,  there  are  multiple  jobs/assignments  within  25A,  and 
due  to  the  mix  and  nature  of  SMEs’  experiences,  we  were  unable  agree  to  a  specific 
job/assignment  within  the  branch  for  discussion.  Differences  in  assignments  frequently  came  up 
as  a  reason  for  differences  in  their  ratings.  19A  officers  tended  to  have  higher  reliabilities  than 
did  the  other  officers.  Based  on  discussions  with  SMEs  and  our  knowledge  of  the  jobs,  it  appears 
that  25A,  3 1  A,  and  88A  are  more  heterogeneous,  with  more  diverse  specialties  and  assignments 
than  19A  officers.  Chapter  III  presents  data  showing  that  25 A  and,  to  a  lesser  extent,  31 A 
yielded  low  reliabilities  on  several  other  descriptor  sets  as  well.  This  lends  some  additional 
credence  to  the  reasons  cited  above  for  lower  reliabilities. 


7  To  ensure  consistency  between  our  data  and  0*NET  data,  we  wanted  to  use  0*NET  formulas  (McCloy,  Waugh,  & 
Medsker,  1998)  for  ICCs.  0*NET  researchers  provided  SPSS™  syntax  that  they  currently  use  for  analyzing  analyst 
ability  ratings.  We  used  that  basic  syntax  for  analyzing  all  of  the  descriptor  sets.  The  formula  ICC  [(3,1)  for  single¬ 
raters  and  (3,  k)  for  multiple  raters]  is  for  a  two-way  mixed-effects  model.  For  readers  most  familiar  with  the 
language  of  McGraw  &  Wong  (1996),  these  ICCs  are  consistency  estimates.  Finally,  occasionally  a  rater  skipped  an 
item.  To  compute  the  ICC,  which  requires  full  data,  we  inserted  the  group  mean  for  the  missing  score. 

8  We  did  discuss  the  ratings  with  SMEs  and  made  edits  to  the  tasks  if  needed  and  even  re-rated  some  of  the  tasks. 
The  ICCs  presented  here  are  based  on  the  original  ratings  of  the  tasks  before  discussion. 

9  These  are  not  necessarily  inordinately  large  numbers  of  raters  for  rating  specific  work-oriented  statements  like 
KWAs.  0*NET  requires  only  8  analysts  for  rating  worker-oriented  abilities  and  skills,  but  those  analysts  receive  2 
Vi  days  of  training  and  practice.  0*NET  requires  larger  numbers  of  incumbent  raters  for  work-oriented  GW  A  and 
work  context  ratings. 


15 


Table  3.  Intraclass  Correlations  (ICCs)  by  Occupation  for  KW A  Ratings 


Occupation 

k 

Number 

of 

KWAs 

Importance  (1-5) 

Frequency  (1-7) 

Performance 
Variability  (1-5) 

ICC 

(3,1) 

ICC 

(3,k) 

ICC 

(3,1) 

ICC 

(3,k) 

ICC 

(3,1) 

ICC 

(3,k) 

31A 

9 

45 

.03 

.23 

.12 

.54 

.06 

.38 

31B 

9 

38 

.16 

.63 

.32 

.81 

.15 

.62 

88A 

5 

28 

.25 

.50 

.42 

.69 

.31 

.57 

88M 

9 

23 

.19 

.68 

.19 

.67 

.03 

.21 

19A 

4 

46 

.29 

.62 

.47 

.78 

.30 

.63 

19K 

6 

37 

.33 

.74 

.46 

.84 

.31 

.73 

25A 

10 

53 

.08 

.45 

.14 

.62 

.04 

.30 

25U 

9 

35 

.24 

.74 

.10 

.50 

.12 

.54 

Total  M 

38.13 

.20 

.28 

.17 

Officer  M 

43.00 

.16 

.29 

.18 

Enlisted  M 

33.25 

.23 

.27 

.15 

Note,  k  =number  of  raters.  ICC  (3,1)  reflects  single  rater  reliability  and  ICC  (3,k)  reflects  average  rater  reliability. 


We  discussed  the  ratings  with  SMEs  and  made  edits  to  the  tasks  if  needed  and  in  a  couple 
of  the  workshops  even  re-rated  some  of  the  tasks.  The  ICCs  presented  here  are  based  on  the 
original  ratings  of  the  tasks  before  discussion. 

While  the  reliabilities  were  lower  than  desired,  the  workshop  KWA  procedures 
accomplished  two  important  goals  for  this  project.  First,  discussing  the  target  occupations  and 
within-occupation  differences  in  KWAs,  SMEs  were  prepared  to  make  ratings  on  the  0*NET 
rating  scales.  Second,  the  KWA  statements,  revised  based  on  the  discussions  could  serve  as 
fodder  for  analysts  to  use  in  making  their  0*NET  ratings. 

We  conducted  a  number  of  analyses  of  the  0*NET  descriptor  ratings.  Specific 
descriptive  statistics  by  occupation  appear  in  a  research  note  published  separately  from  this 
report  (Russell  et  al.,  2008).  The  reliabilities  and  other  relevant  data  are  summarized  and 
discussed  in  Chapter  III. 


Analyst  Data  Collection 

As  mentioned  at  the  beginning  of  this  chapter,  two  sets  of  analysts  were  included  in  the 
research,  project  analysts  and  0*NET  analysts.  Both  sets  of  analysts  participated  in  the  same 
training  program  to  learn  to  make  ratings  on  the  0*NET  abilities  and  skills. 

S/imu/us  Materia/s /or  t/ie  Ratings 

Over  the  years,  the  stimulus  materials  for  each  occupation  being  rated  have  been 
standardized  (Donsbach  et  al.,  2003).  For  each  of  the  eight  Army  target  occupations,  we  prepared 
stimulus  materials  mirroring  those  used  in  the  0*NET  system  for  collecting  analyst  ratings  on 
civilian  occupations.  These  materials  included: 
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•  Title  and  definition  of  the  occupation 

•  Task  sheet:  A  list  of  tasks  that  were  rated  by  Army  SMEs  in  terms  of  their 
importance  to  the  job.10  The  tasks  were  categorized  as: 

o  Core  tasks  if  they  received  a  mean  SME  importance  rating  of  >  3.0. 

o  Supplementary  tasks  if  they  received  a  mean  SME  importance  rating  of  < 
3.0. 

•  Ability  sheets:  A  sheet  for  each  of  the  52  abilities.  The  following  information  was 
contained  on  each  ability  sheet: 

o  The  definition  of  the  ability  was  listed  at  the  top  of  the  sheet. 

o  An  ability  level  scale  with  level  scale  anchors  was  provided  directly 
beneath  the  definition  of  the  ability  in  order  to  facilitate  the  analysts’ 
understanding  of  the  ability. 

o  GW  As  that  were  (a)  linked  to  the  given  ability* 1 1  and  (b)  received  an  SME 
mean  importance  rating  of  >_3.0. 

o  Work  Context  variables  (WCs)  that  were  (a)  linked  to  the  given  ability 
(same  linkage  criteria  as  for  GW  As)  and  (b)  received  a  mean  SME  rating  of 
>_3.0. 

•  Skill  sheets:  A  sheet  was  provided  for  each  of  the  35  skills.  The  following 
information  was  contained  on  each  skill  sheet: 

o  The  definition  of  the  skill  was  listed  at  the  top  of  the  sheet. 

o  A  skill  level  scale  with  level  scale  anchors  was  provided  directly  beneath 
the  definition  of  the  skill  in  order  to  facilitate  the  analysts’  understanding 
of  the  skill. 

o  GWAs  that  were  (a)  linked  to  the  given  skill  and  (b)  received  an  SME 
mean  importance  rating  of  >_3.0. 

o  Work  Context  (WC)  variables  that  were  (a)  linked  to  the  given  skill  and  (b) 
received  a  mean  SME  rating  of  >_3.0. 

These  stimulus  materials  matched,  as  closely  as  possible,  the  materials  used  by  0*NET 
analysts  for  civilian  occupations.  The  KWAs  that  came  out  of  the  Army  SME  workshops  were 
used  as  the  tasks,  and,  where  needed,  minor  wording  changes  were  made  to  remove  military- 
specific  language.  Additionally,  we  prepared  a  glossary  of  Army  terms  for  the  analysts  to 
reference  as  needed. 


10  Since  0*NET  analysts  are  accustomed  to  the  term  tasks,  we  called  KWAs  tasks  for  the  purpose  of  the  analyst 
ratings. 

11  The  linkage  judgments  were  made  by  experienced  industrial/organizational  psychologists  in  a  prior  study  (see 
Donsbach  et  al.,  2003  for  a  description  of  the  linkage  study).  Essentially,  the  judgments  were  made  by  asking  whether 
a  given  ability  is  required  to  perform  a  particular  GWA. 
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The  analysts  used  the  ability  and  skill  definitions  and  rating  scales  in  the  0*NET 
questionnaires  to  make  their  ratings.  These  questionnaires  are  the  same  ones  used  in  the  SME 
workshops.  A  sample  importance  and  level  rating  scale  appears  in  Figure  5. 

Overview  of  t/ie  Training 

0*NET  researchers  allowed  us  to  carry  out  our  data  collection  efforts  in  concert  with 
their  ongoing  0*NET  data  collection  activities.  The  0*NET  analyst  rater  training  follows  an 
analyst  training  manual  that  includes  four  training  modules: 

Module  1:  History  of  0*NET.  This  module  generally  describes  the  background 
associated  with  0*NET,  as  well  as  what  information  0*NET  Online  contains.  In 
addition,  the  following  five  rating  tips  are  provided: 

•  Do  not  hesitate  to  use  the  extremes  (1  and  5  on  the  Importance  scales;  1  and  7  on  the 
Level  scales)  when  assigning  the  ratings.  If  you  avoid  using  the  extremes,  you  reduce 
the  scales  to  fewer  levels  than  intended. 

•  Do  not  make  all  ratings  at  one  end  of  the  rating  scale.  Check  yourself  to  ensure  your 
ratings  do  not  cluster  only  at  the  high  end  or  only  at  the  low  end  of  the  scale. 

•  It  is  possible  that  you  have  formed  preconceived  ideas  of  the  definition  of  a  given 
occupation  or  descriptor.  However,  it  is  very  important  that  you  do  not  rely  on  your 
personal  knowledge  or  experience  to  make  importance  and  level  ratings.  Rather,  you 
should  use  only  the  information  provided  in  the  stimulus  materials  to  make  these 
ratings. 

•  Be  sure  to  consider  all  of  the  occupational  information  provided  in  the  stimulus 
materials  to  make  your  ratings.  For  example,  information  on  several  descriptors 
relevant  to  the  target  ability  or  skill  is  presented  in  the  stimulus  materials.  As  you  will 
be  trained,  information  considered  at  the  end  of  the  rating  process  for  a  given  ability 
or  skill  may  influence,  or  even  change,  your  initial  rating. 

•  Rate  abilities  and  skills  independent  of  one  another.  That  is,  do  not  let  your  rating  of 
one  ability  or  skill  influence  the  ratings  of  other  abilities  or  skills  within  a  given 
occupation.  You  should  begin  a  new  rating  process  with  each  ability  and  skill  you 
rate.  In  addition,  do  not  let  your  ratings  from  one  occupation  impact  your  ratings  for 
another  occupation. 

Approximately  30  minutes  are  allotted  for  Module  1. 

Module  2:  Overview  of  the  Stimulus  Materials.  In  this  module  analysts  review  and 
discuss  the  stimulus  materials  that  are  provided  to  the  analysts  for  making  their  ability 
and  skill  ratings.  Analysts  go  over  the  definitions  for  each  of  the  abilities,  skills,  GW  As 
and  WCs.  Analysts  also  review  and  discuss  the  various  rating  scales  on  which  the  above 
constructs  are  rated.  Approximately  1  hour  and  30  minutes  are  allotted  for  Module  2. 
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Module  3:  Making  Ratings.  In  this  module  a  step-by-step  description  of  the  ability  and 
skill  rating  process  is  presented.  The  same  process  is  followed  for  making  both  ability 
and  skill  ratings.  Time  is  also  spent  making  practice  ratings  on  several  of  the  0*NET 
abilities  and  skills  for  a  sample  of  0*NET  occupations.  Approximately  6  hours  are 
allotted  for  Module  3.  The  basic  steps  for  making  an  ability/skill  rating  are  as  follows: 

•  Step  1 :  Review  the  occupation  title,  definition,  and  tasks  to  get  a  full  picture  of  the 
occupation. 

•  Step  2:  Review  the  title,  definition,  and  level  scale  anchors  of  the  construct  (i.e.,  the 
ability/skill  in  question)  you  are  about  to  rate. 

•  Step  3:  Review  the  tasks  and  their  importance  ratings.  Focus  on  the  core  tasks  first, 
followed  by  the  supplementary  tasks.  Think  of  a  preliminary  importance  rating  for 
this  construct. 

•  Step  4:  Review  the  linked  GWAs  and  their  mean  importance  ratings.  If  necessary, 
adjust  your  preliminary  importance  rating  for  this  construct. 

•  Step  5:  Review  the  linked  WC  statements  and  their  mean  ratings.  If  necessary,  adjust 
your  importance  rating  for  this  construct. 

•  Step  6:  Document  your  rating  for  this  construct.  If  your  importance  rating  was  >  2, 
then  make  a  level  rating  for  this  construct  (repeat  Steps  1  -  6). 

Module  4:  Recording  Your  Ratings.  In  this  module,  the  analysts  were  introduced  to  the 
electronic  rating  form  that  they  would  be  using  for  entering  their  ability  and  skill  ratings. 
Three  hours  were  allotted  for  Module  4. 

Each  of  the  four  training  module  incorporates  hands-on  exercises  and  quizzes.  In 
addition,  there  is  a  manual  for  the  trainers  with  instructions  for  presenting  information. 

The  standard  0*NET  analyst  training  modules  described  above  are  spread  out  over  two 
days  of  training.  We  developed  and  administered  an  additional  half  day  of  training  (after 
completion  of  the  first  two  days)  to  acquaint  raters  with  this  evaluation  project,  the  Army,  Army 
occupations,  and  military  terminology.  The  half-day  session  was  a  mix  of  lecture  and  hands-on 
practice. 

For  15  of  the  16  0*NET  analysts,  this  was  refresher  training;  they  had  been  trained 
before  and  were  excused  from  the  first  half  day  of  training  covering  the  history  of  0*NET  and 
description  of  the  stimulus  materials.  All  of  the  project  analysts  and  one  of  the  0*NET  analysts 
were  newcomers  to  the  0*NET  training  and  attended  the  full  2  Vi  days. 

dr  my- Wide  and  M OS-Specific  Tasks 

Initially,  we  had  planned  to  ask  raters  to  consider  Army-wide  tasks  (i.e.,  KWAs)  in 
addition  to  the  MOS-specific  KWAs,  and  we  had  prepared  an  Army-wide  task  list  for  analysts  to 
use  in  making  their  ratings.  During  the  workshop,  we  decided  to  omit  the  Army-wide 
information  for  several  reasons.  First,  there  were  no  importance  ratings  for  Army-wide  lists  like 
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we  had  for  the  occupation-specific  task  lists  (i.e.,  KWAs).  Consequently,  the  analysts  expressed 
confusion  about  how  to  factor  in  the  Army-wide  list  into  their  ratings.  Second,  there  was  a 
sentiment  that  if  all  Army  occupations  were  rated  on  these  common  tasks,  then  the  ability  and 
skill  ratings  for  the  various  Army  occupations  would  end  up  looking  quite  similar  (i.e.,  using  the 
Army-wide  task  list  might  blur  the  distinction  between  the  ability  and  skill  ratings  across 
occupations).  Third,  an  ARI  analyst  indicated  that  ARI  generally  does  not  use  an  Army-wide 
task  list  to  describe  specific  occupations.  Finally,  having  the  analysts  factor  in  the  Army-wide  list 
along  with  the  job-specific  task  list  (and  the  WCs  and  the  GWAs)  was  seen  by  the  analysts  as 
being  overly  burdensome.  For  these  reasons,  we  decided  to  omit  the  Army-wide  task  list  from 
the  analysts’  final  stimulus  materials. 

/fating  Assignments  anc/  Jfesu/ts 

The  eight  project  analysts  each  made  ability  and  skill  ratings  on  all  8  target  occupations. 
Half  (8)  of  the  0*NET  analysts  made  ratings  for  31  A,  3  IB,  88  A,  and  88M.  The  other  half  (8) 
made  ratings  for  1 9A,  1 9K,  25A,  and  25U.  Descriptive  statistics  for  the  project  and  0*NET 
analyst  data  appear  alongside  SME  results  in  a  research  note  published  separately  from  this 
report  (Russell  et  al,  2008).  Chapter  III  presents  reliabilities  and  other  statistics  for  the  analyst 
ratings. 
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III.  Is  0*NET  Sufficient? 


Chapter  I  posed  the  following  question  for  the  evaluation:  Is  the  0*NET  taxonomy 
sufficient  for  analyzing  Army  occupations?  A  complete  answer  to  this  question  is  not  possible  in 
a  single  research  effort  of  this  scope,  but  the  basic  psychometric  and  “content  coverage”  issues 
can  be  addressed  for  some  of  the  0*NET  descriptor  domains  that  appear  to  be  most  useful  for 
the  Army.  Accordingly,  we  made  comparisons  between  Intraclass  Correlation  Coefficients 
(ICCs)  and  mean  descriptor  profiles  using  the  data  collected  in  SME  and  analyst  workshops 
described  in  Chapter  II.  2  Those  data  included  ratings  for  eight  Army  occupations — 31  A,  3 IB, 
88A,  88M,  19A,  19K,  25A,  and  25U — on  four  of  the  0*NET  occupation  descriptor  domains — 
GW  As,  work  context,  abilities,  and  skills — made  by  three  different  types  of  raters — Army 
SMEs,  project  analysts  and  0*NET  analysts. 

Using  these  data,  the  following  key  evaluation  questions  are  addressed  in  this  chapter: 

•  Are  Army  occupations  rated  reliably  using  0*NET  rating  scales? 

•  Do  ratings  on  0*NET  rating  scales  differentiate  Army  occupations? 

•  How  well  do  SME  and  analyst  ratings  of  Army  jobs  agree? 

•  Are  ratings  on  Army  occupations  similar  in  quality  to  ratings  on  civilian  occupations? 

•  Are  Army  occupation  profiles  similar  to  those  for  their  civilian  counterparts? 

Are  Army  Occupations  Rated  Reliably  Using  0*NET Rating  Scales? 

To  examine  the  reliability  of  the  ratings,  we  calculated  the  intraclass  correlations  (ICC  [3, 

1]  and  [3,  &];  Shrout  &  Fleiss,  1979)  among  the  analysts’  ratings  to  look  at  consistency  across 
constructs  within  occupations.13  The  0*NET  target  level  of  multi-rater  reliability  is  a  median  ICC 
(3,  k)  of  .80  or  greater.  The  value  of  .80  has  been  judged  to  be  a  good  rule-of-thumb  that  has  been 
used  previously  in  the  0*NET  context  (e.g..,  McCloy,  Waugh,  &  Medsker,  April  1998). 

A£>tZ//ies  and SC ids 

As  shown  in  Table  4,  the  SME  ICCs  for  abilities  are  about  20-30  points  lower  than  those 
for  the  analyst  groups.  SME  ratings  of  officer  occupations  yielded  ICCs  that  were  on  average 
slightly  lower  than  those  for  enlisted  occupations.  SME  ability  level  ratings  for  3 1  A,  3 1 B,  and 
19K  reached  the  target  ICC  of  .80,  even  though  SMEs  did  not  receive  the  2  Vi  day  training 
program  that  analysts  received.  The  judgments  from  eight  project  and  eight  Q*NET  analysts 
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Our  analyses  were  somewhat  dictated  by  available  0*NET  information.  While  a  wealth  of  data  are  available  at 
http://online.onetcenter.org/  for  0*NET,  the  bulk  of  the  reports  with  statistics  of  interest  to  us  (e.g.,  ICCs)  were  only 
available  for  abilities. 

13  To  ensure  consistency  between  our  data  and  0*NET  data,  we  wanted  to  use  0*NET  formulas  (McCloy,  Waugh, 

&  Medsker,  1998)  for  ICCs.  0*NET  researchers  provided  SPSS™  syntax  that  they  currently  use  for  analyzing 
analyst  ability  ratings.  We  used  that  basic  syntax  for  analyzing  all  of  the  descriptor  sets.  The  formula  ICC  [(3,1)  for 
single-raters  and  (3,  k)  for  multiple  raters]  is  for  a  two-way  mixed-effects  model.  For  readers  most  familiar  with  the 
language  of  McGraw  &  Wong  (1996),  these  ICCs  are  consistency  estimates.  Finally,  occasionally  a  rater  skipped  an 
item.  To  compute  the  ICC,  which  requires  full  data,  we  inserted  the  group  mean  for  the  missing  score. 
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were  sufficient  to  reach  the  recommended  .80.  There  appear  to  be  no  reliability  differences 
between  the  importance  and  level  scales  on  abilities. 

For  skills,  the  analyst  ICCs  were  about  ten  to  twenty  points  higher  than  the  SME  ICCs  for 
the  single-rater  reliabilities  (see  Table  5),  but  all  groups  of  raters  reach  acceptable  levels  for  both 
the  importance  and  level  scales.  All  rater  types  achieved  higher  ICCs  on  the  skills  than  on  the 
abilities,  with  exception  of  the  25A  SME  ratings. 

It  is  important  to  remember  that  SMEs  received  only  basic  introduction  to  the  ability  and 
skills  ratings,  and  both  sets  of  analysts  received  extensive  training.  When  SMEs  discussed  their 
abilities  and  skills  ratings,  they  tended  to  agree  with  the  rank  ordering  of  abilities  and  skills 
based  on  their  ratings.  When  we  drew  their  attention  to  items  with  larger  SDs,  comments  usually 
had  to  do  with  the  applicability  of  the  anchors  (especially  level  anchors)  to  Army  jobs,  or  their 
occupations,  in  particular.  Some  SMEs  interpreted  the  anchors  too  literally  or  otherwise 
experienced  difficulty  translating  the  anchors  to  their  job. 

Another  general  theme  was  that  some  of  the  larger  SDs  on  abilities  were  for  psychomotor 
and  sensory  abilities.  SMEs  felt  like  some  of  the  psychomotor  variable  names  were  hard  to 
understand  (e.g.,  Dynamic  Flexibility)  and  to  differentiate  (e.g.,  Explosive  Strength  and  Dynamic 
Strength).  For  some  sensory  abilities,  particularly  Night  Vision,  SMEs  were  unsure  how  to  make 
ratings  given  that  they  have  assistive  devices  (e.g.,  flashlights,  goggles).  These  are  the  types  of 
topics  that  are  covered  in  the  analyst  training,  and  if  the  Army  were  to  use  0*NET  “as  is”  and 
rely  on  SME  raters,  they  could  also  benefit  from  training  of  that  nature. 

Six  of  the  16  0*NET  analysts  commented  on  rating  the  Army  jobs.  Four  of  the  six  stated 
that  they  thought  0*NET  was  useful  for  describing  the  Army  jobs.  The  other  two  provided 
suggestions  for  information  to  give  analysts  in  the  future. 

Work  Con  (ex/ 

ICCs  for  work  context  ratings  appear  in  Table  6.  As  shown,  the  multi-rater  ICCs  show 
acceptable  levels  of  reliability  for  all  but  88A,  which  had  only  five  raters. 

There  was  much  discussion  about  the  work  context  variables  in  most  of  the  SME 
workshops,  in  part  prompted  by  us.  Prior  to  the  workshops  we  had  noted  that  work  context  might 
need  to  change  somewhat  for  Army  occupations  and,  therefore,  we  did  more  probing  for  what 
might  be  missing  or  what  might  need  to  change  in  this  descriptor  set. 

The  main  issues  that  emerged  were: 

•  SMEs  indicated  that  the  work  context  ratings  were  affected  by  the  deployment  or 

garrison  assignment.  For  example,  work  hours  during  some  deployments  are  24  hours  a 
day  seven  days  a  week.  A  day  off  during  deployment  might  be  a  day  doing  maintenance 
instead  of  patrolling.  Similarly,  the  physical  requirements  are  generally  higher  for 
deployments.  Some  SMEs  suggested  that  they  should  have  made  ratings  separately  for 
deployments  and  in-garrison  activities. 
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Table  4.  ICCs  for  Ability  Importance  and  Level  Ratings  by  Occupation  and  Rater 
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Table  6.  ICCs  by  Occupation  for  Work  Context  Ratings 


ICC  ICC 

Occupation  n (3,1) (3,k) 


31A 

9 

.47 

.89 

31B 

9 

.50 

.90 

88A 

5 

.48 

.73 

88M 

9 

.36 

.84 

19A 

4 

.58 

.84 

19K 

6 

.64 

.91 

25A 

10 

.44 

.89 

25U 

9 

.46 

.88 

Total  M 

^49 

Officer  M 

.49 

Enlisted  M 

.49 

Note.  ICC  (3,1)  reflects  single  rater  reliabilities  and  ICC  (3,k)  reflects  average  rater  reliabilities.  Mean  ICCs  are  not 
provided  for  multi-rater  Army  SMEs  estimates  because  the  number  of  raters  varied  across  occupations. 


•  Some  Army-specific  work  context  variables  are  probably  needed  to  address  the  following 
issues: 


■  Work  Pace.  One  WC  item  asks  how  important  is  it  to  keep  a  pace  set  by 
machinery  or  equipment.  It  prompted  a  discussion  about  how  work  pace  varies  for 
combat  and  non-combat  activities..  During  battle,  the  pace  is  influenced  by 
available  technology  as  well  as  other  factors  (e.g.,  own  and  enemy  troop  strength). 

■  Working  with  little  sleep.  The  discussion  of  work  hours  also  prompted  discussions 
about  sleep  deprivation.  SMEs  said  that  sleep  deprivation  affects  the  mission  and 
is  influenced  by  the  mission.  It  appears  that  junior  officers,  in  particular,  often 
operate  with  very  little  sleep.  It  would  seem  that  the  need  to  work  with  little  sleep 
could  be  an  important  context  element  to  capture. 

■  Communicating  with  indigenous  people,  host  nation  counterparts,  peers  and 
superiors.  Another  theme  that  tended  to  come  from  officers  had  to  do  with 
communications.  During  deployments,  junior  officers  have  been  assigned  to  work 
with  host  nation  counterparts  (e.g.,  Iraqi  junior  officers)  and  to  negotiate  with 
indigenous  leaders  about  activities  and  resources.  While  much  of  the  negotiation 
takes  place  through  an  interpreter,  it  is  still  a  challenge  to  deal  with  the  cultural 
differences  in  the  negotiation.  They  also  thought  something  could  be  added  to 
reflect  their  need  to  persuade  or  influence  their  peers  or  superiors. 

■  Travel,  overseas  travel,  and  extended  time  away  from  home.  SMEs  mentioned 
that  items  could  deal  with  being  separated  from  one’s  normal  environment  (e.g., 
in  a  foreign  country)  for  an  extended  period  of  time. 

■  Lifting  heavy  weight.  Some  SMEs  felt  that  lifting  heavy  weight  was  not 
sufficiently  covered  in  the  WC  items. 

•  SMEs  suggested  that  Army  examples  could  be  added  to  some  items  to  make  them 
clearer.  For  example,  a  WC  item  about  protective  gear  could  add  examples  of  combat 
gear  to  the  list  of  civilian  gear  in  that  item. 
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Genera/izecf  fV ork  Activities 


As  shown  in  Table  7,  all  of  the  occupations  have  acceptable  levels  of  reliability,  except 
for  25A.  (MOS  88A  and  19A  had  low  numbers  of  raters,  but  would  reach  the  .80  level  of 
reliability  if  at  least  6  raters  were  used).  GWA  ratings,  like  those  for  work  context  and  KWAs, 
are  influenced  by  variability  within  the  jobs  and  across  deployment  experiences  of  SMEs.  As 
described  in  Chapter  II,  work  variability  was  an  issue  for  several  jobs  and  for  25A  in  particular. 

Table  7.  ICCs  by  Occupation  for  Generalized  Work  Activity  Ratings 


Occupation 

n 

Importance 

Level 

ICC 

(3,1) 

ICC 

(3,k) 

ICC 

(3d) 

ICC 

(3,k) 

31A 

9 

.50 

.90 

.56 

.92 

31B 

9 

.57 

.92 

.63 

.94 

88A 

3 

.43 

.69 

.48 

.73 

88M 

9 

.42 

.87 

.42 

.87 

19A 

4 

.40 

.73 

.50 

.80 

19K 

6 

.58 

.89 

.61 

.90 

25A 

10 

.25 

.77 

.19 

.70 

25U 

9 

.41 

.86 

.45 

.88 

Total  M 

.45 

.48 

Officer  M 

.40 

.43 

Enlisted  M 

.50 

.53 

Note.  ICC  (3,1)  reflects  single  rater  reliabilities  and  ICC  (3,k)  reflects  average  rater  reliabilities.  Mean  ICCs  are  not 
provided  for  multi-rater  Army  SMEs  estimates  because  the  number  of  raters  varied  across  occupations. 


As  they  did  with  the  ability  and  skills  scales,  SMEs  tended  to  agree  with  the  rank 
ordering  of  GWAs  based  on  their  mean  ratings.  When  we  prompted  them  to  discuss  GW  As  with 
high  SDs,  the  main  issue  was  the  applicability  of  the  level  anchors.  For  instance,  the  level 
anchors  for  “Inspecting  equipment,  structures,  or  materials”  -  “Check  that  doors  to  building  are 
locked,”  “Inspect  equipment  in  a  chemical  processing  plant,”  and  “Inspect  a  nuclear  reactor”  - 
required  too  great  of  an  inference  for  some  SMEs  to  be  certain  of  a  rating.  Even  so,  they  were 
able  to  make  their  judgments  reliably. 

Do  Ratings  on  0*NET  Rating  Scales  Differentiate  Army  Occupations? 

Differentiation  among  occupations  is  very  important  to  the  Army,  as  it  is  needed  for 
accurate  classification  of  recruits  into  occupations  based  on  their  abilities  and  skills.  Tables  4  and 
5  showed  that  0*NET  analysts  rated  the  abilities  and  skills  more  reliably  than  did  SMEs  and 
project  analysts,  albeit  the  differences  between  0*NET  and  project  analyst  ICCs  were  small. 

But,  it  is  possible  that  the  analysts  agree  with  each  other  but  do  not  pick  up  on  important 
differences  among  occupations. 

We  used  a  multi-trait  multi-method  approach  (Campbell  &  Fiske,  1959)  to  address  this 
issue.  High  correlations  among  different  types  of  raters  for  the  same  occupation  (e.g.,  correlation 
between  31 A  ratings  made  by  SMEs  and  project  analysts)  would  indicate  convergent  validity. 
That  is,  different  methods  (i.e.,  rater  types)  yield  converging  results  for  an  occupation.  If  these 
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convergent  validity  correlations  are  higher  than  the  correlation  between  different  occupations 
rated  by  the  same  rater  type  (e.g.,  the  correlation  between  31 A  and  19K  ratings  made  by  project 
analysts),  there  is  evidence  of  discriminant  or  divergent  validity.  Divergent  validity  for  a  rater 
type  would  indicate  that  the  raters  are  differentiating  occupations. 

To  assess  convergence  and  divergence,  we  computed  correlations  among  mean  ability 
level  vectors  by  occupation  and  analyst  type,  as  shown  in  Table  8,  and  did  the  same  for  mean 
skill  level  vectors  (Table  9).  Table  10  summarizes  across  Tables  8  and  9.  We  also  investigated 
divergence  for  WCs  and  GWAs,  where  we  had  no  separate  rater  types  to  compare.  Those  data 
are  discussed  after  abilities  and  skills. 

Convergent  Corre/ations  based  on  Abi/ity  and  S/c id L  eve/ datings 

Table  1 0  illustrates  three  main  points  regarding  convergent  correlations: 

•  First,  regarding  analyst  type,  project  and  0*NET  analyst  means  correlate  very  highly 
(mean  r  =  .79  for  abilities  and  .92  for  skills  across  all  occupations).  SME  and  project 
analysts  ratings  correlated  more  highly  than  SME  and  0*NET  analyst  ratings.  SME 
ratings  were  less  reliable  than  the  other  two.  Even  so,  correcting  for  unreliability  would 
not  change  the  fact  that  SME  and  0*NET  analyst  ratings  correlate  less  than  SME  and 
project  analysts  because  the  reliabilities  for  the  0*NET  and  project  analysts  were  almost 
the  same. 

•  Second,  for  abilities,  the  convergent  correlations  were  higher  for  enlisted  occupations 
than  officer  occupations  (rs  =  .53,  .36,  and  .77  for  officer  compared  to  .60,  .45,  and  .81 
for  enlisted).  This  suggests  that  the  three  rater  types  had  a  better  and  more  common 
understanding  of  the  ability  requirements  of  the  enlisted  occupations. 

•  Third,  convergent  correlations  were  higher  for  skills  than  for  abilities. 

Divergent  Corre/ations  Based  on  Abi/ity  and  S/c id L  eve/ B a  tings 

Three  points  can  also  be  made  with  regard  to  the  divergent  correlations  in  Table  1 0: 

•  The  0*NET  analyst  ratings,  on  average,  differentiated  less  between  occupations  than  did 
either  the  SME  or  the  project  analyst  ratings.  Specifically,  the  0*NET  analysts  mean 
ratings  for  different  occupations  were  correlated  .60  (abilities)  and  .65  (skills)  on  average. 
In  contrast,  SME  and  project  analyst  mean  ratings  for  different  occupations  correlated 
between  .44  and  .59,  indicating  more  differentiation,  particularly  for  the  abilities.  It  is 
possible  that  SMEs  and  project  analysts  understood  the  KWAs  better  and  could  therefore 


27 


m 


a 

« 

C 

•« 

!• 

o 

"C* 

5*5 

z 

•« 

■S 

e 

a 

i 


c 

o© 

-3 

-c 

£ 


b  a, 

«d  £ 
C£  H* 


Os  ^ 
Oo  ON 


NO  *-N 

OQ  SO 


00  wm  o> 
tt  n 


N 

Oo 


*''  IT) 

os  o* 


*■<  00 
rf  ^ 


K 

.37 

ON 

CD 

*n 

04 

o 

Os 

oo 

OO 

«N 

CD 

m 

sO 

Os 

ON 

oq 

oq 

Os 

CD 

NO 

so 

oc 

rr 

NO 

NO 

in 

04 

CD 

CD 

Oo 

m 

CD 

TT 

CD 

O' 

© 

oq 

o 

04 

m 

04 

t}; 

oo 

04 

04 

ON 

CD 

04 

O 

oo 

00 

oo 

O 

i* 

O 

r 

© 

i* 

in 

© 

p 

04 

r- 

04 

O' 

ON 

in 

oo 

OO 

SO 

• 

CD 

CD 

i 

m 

o 

sn 

m 

wmm 

© 

o 

04 

oo 

Os 

04 

i 

CD 

m 

m 

O' 

O' 

m 

m 

O' 

m 

© 

m 

oo 

o 

Tt 

°9 

O; 

o 

SO 

in 

CD 

rr 

04 

04 

04 

NO 

so 

ON 

o 

in 

. 

m 

o 

04 

O' 

o 

rv 

SO 

CD 

O' 

m 

m 

• 

rr 

o 

Os 

, 

_ 

CD 

04 

m 

o 

oo 

04 

00 

SO 

sO 

CD 

O' 

oo 

04 

CD 

CD 

CD 

in 

oo 

04 

rf 

ON 

ON 

so 

m 

sO 

m 

rt 

— , 

Os 

O; 

sO 

O' 

tJ- 

oo 

O' 

o 

o 

© 

“ 

90 

cd 

O 

»n 

O' 

ON 

04 

CD 

O' 

sO 

NO 

NO 

NO 

m 

SO 

sO 

rr 

Tf 

* 

O'  o  —  O' 

—  —  in  — 


m  On 

Tt  O' 


On  m  ~ 
04  ©  m 


(N  m  m  O'  04  in 

U~)  0C  OO  O  f  O' 


CD  OO  SO  O 
CD  ID  *D  SO 


O'Tj-oi  —  O'  —  ooi^vomm 

OOOO  —  Tj-Tj-sOmm—  m  “ 


-  ^  Tf  (N 

CD  04  OJ  m 


soinTfoiinoNO-  —  rtmso 
coo4roo404mo404cno4  — 


—  OO  O' 

ro  nj-  in 


so  (N  Nt  - 

rN  m  x  in 


^m^niNOTf^iNtso^ 

Tj-u^CNNOsOtNromrOTfT^ 


CD  —  ON 
TtsD  (N 


—  — •  o*  r- 
id  O'  so  so 


mr^mcnoiojoomoooi 

so  Tf  m  so  in  m  —  —  cd  ’ —  — 


m  (N  O 
sf  (N  - 


On  Tfr  ©  04 

sO  NO  *D  O' 


ONONONfoONoor^-r-o  — 

m  o  so  r-  Tf  o  —  cm  Tt  —  — 


04  sO 

id  r- 


OO  SO  O 
(N  so  ON 


SO  CD  O  NO 
O  04  NO  04 


0N0'f0O4TfiO»O  —  004^ 

nj-  h  fs  -  roior-r-ososo 


—  04  CD  — 


OQ 


4  sO 

04 

O' 

in 

O 

On 

ON 

m 

© 

OO 

CD 

ON 

04 

in 

O' 

m 

04 

m 

04 

3  O' 

sO 

00 

sO 

04 

m 

sO 

O' 

o 

1 

CD 

CD 

m 

m 

CD 

CD 

N  CD 

O' 

04 

m 

CD 

ON 

CD 

OO 

CD 

00 

O' 

04 

CD 

-.04 

ON 

O 

r- 

'  sO 

m 

CD 

© 

m 

m 

Dt 

sO 

D- 

in 

CD 

04 

o 

o 

1 

o 

i 

<  04 

CD 

- 

04 

CD 

- 

04 

CD 

- 

04 

CD 

- 

04 

CD 

- 

04 

CD 

- 

CM 

CD 

1 

< 

OQ 

< 

< 

oo 

oo 

ON 

On 

m 

m 

) 

oc 

oo 

— 

04 

04 

Mote.  Rater  types  were  as  follows:  1  =  SME  raters.  2  =  Project  Analysts,  3  =  0*NET  Analysts.  Reliabilities  (ICC  [3,k])  appear  on  the  diagonal.  Same-occupation 
correlations  are  in  bold. 


25A  25U 

m 

(N 

m 

fN 

.88 

.83  .91 

.81  .87  .93 

.55 

.66  .86 

.54  .84  90 

.47  .54  .46 

.35  .57  .44 

.39  .56  .55 

m 

m 

Tt 

m 

Os 

Os 

rn 

in 

in 

ft 

ft 

Os 

If) 

o 

r- 

m 

o 

oo 

oo 

OS 

Ov 

Os 

m 

m 

ft 

ft 

oo 

m 

00 

so 

__ 

OS 

Os 

SO 

m 

fN 

fN 

ft 

sO 

so 

SO 

Os 

Os 

fN 

O 

o 

Os 

Os 

m 

m 

m 

r- 

00 

m 

m 

< 

<N 

Tf 

, 

, 

Os 

m 

m 

so 

m 

SO 

fN 

OS 

Os 

Os 

fN 

sn 

rl- 

m 

00 

m 

fN 

SO 

m 

rf 

oo 

V~) 

in 

so 

m 

SO 

Oo 

ro 

m 

SO 

m 

m 

m 

fT) 

> 

o 

Tf 

00 

OO 

«n 

fN 

m 

m 

fN 

fN 

Os 

SO 

so 

m 

sn 

oo 

OO 

m 

m 

m 

m 

in 

ft 

*> 

ft 

2 

(N 

Os 

ft 

t 

O 

fN 

fN 

m 

m 

fN 

"t 

fN 

fN 

Sr 

oo 

oo 

oc 

in 

iT) 

Tf 

Os 

oo 

m 

in 

sO 

ft 

OO 

oo 

k. 

so 

IT) 

in 

ft 

Tf 

SO 

fN 

OS 

oo 

m 

SO 

fN 

ft 

& 

00 

SO 

m 

Tf 

ro 

fN 

00 

SO 

IT) 

m 

m 

m 

ft 

SO 

in 

q 

as 

ro 

, 

t 

Os 

SO 

l/S 

fN 

so 

O 

o 

fN 

so 

SO 

oo 

fN 

"T3 

Os 

m 

SO 

ft 

Os 

Os 

fN 

m 

*r> 

oo 

m 

fN 

Tf 

s: 

< 

(N 

‘■n 

Tt 

m 

ft 

fN 

sn 

*T) 

fN 

O 

SO 

o 

r- 

Os 

£ 

oo 

Os 

Os 

in 

SO 

oo 

Os 

Os 

fN 

iri 

in 

m 

oo 

oo 

m 

•2 

oc 

Q 

, 

m 

ft 

o 

00 

00 

m 

00 

O 

fN 

Os 

o 

O 

o 

oo 

°o 

ft 

SO 

m 

fN 

m 

oo 

so 

SO 

— 1 

fN 

in 

r- 

in 

m 

Sr 

Cj 

CJ 

ro 

r\ 

00 

_ 

m 

m 

Tf 

in 

oo 

Os 

m 

l^s 

,<fr 

Os 

oo 

fN 

in 

© 

Os 

m 

OO 

oo 

SO 

oo 

ft 

m 

SO  s£) 

m 

«n 

in 

m 

m 

in 

■c 

OQ 

(N 

vo 

m 

ft 

fN 

so 

»n 

m 

r— 

SO 

m 

fN 

Tf 

fN 

o 

OS 

ft 

ft 

m 

Os 

Os 

SO 

OO 

00 

rf 

in 

ft 

OO 

r- 

r- 

m 

m 

in 

m 

in 

m 

m 

fN 

m 

•2 

» n 

00 

SO 

Tf 

m 

ft 

m 

m 

ft 

o 

fN 

O 

o 

Tf 

m 

ft 

sO 

'W 

Q 

Os 

SO 

so 

cn 

Tf 

ft 

ft 

ft 

m 

m 

r- 

m 

so 

in 

sO 

*** 

S 

r*^ 

1\ 

OO 

o 

oo 

oo 

Os 

o 

V— 4 

ft 

Os 

m 

*—i 

m 

,^- 

oo 

oo 

1—4 

fN 

Os 

SO 

m 

Os 

fN 

oo 

ft 

so 

OO 

Os 

fN 

fN 

Os 

os 

o 

fN 

fN 

SO 

ft* 

O 

fN 

© 

■*«* 

rsi 

SO 

Os. 

fN 

oo 

m 

os 

Os 

ft 

Os 

oo 

so 

i—i 

r- 

»n 

m 

in 

oo 

OO 

OO 

s 

< 

Os 

Os 

fN 

OO 

ft 

ft 

oo 

00 

fN 

Tf 

OO 

00 

o 

fN 

fN 

SO 

SO 

o 

fN 

m 

i 

o 

in 

_ 

m 

rf 

m 

ft 

m 

o 

SO 

sn 

oo 

r^ 

—1 

OO 

•— 1 

Os 

Os 

oo 

1— < 

— ■* 

OO 

*l5J 

Os 

00 

ft 

ft* 

SO 

OO 

ft 

SO 

m 

(N 

m 

00 

so 

SO 

*- 1 

in 

fN 

O 

o 

-* 

o\ 

— 

fN 

m 

— 

fN 

m 

— 

fN 

m 

— 

fN 

m 

fN 

m 

fN 

m 

— 

fN 

m 

fN 

m 

■*«» 

-C 

< 

CQ 

< 

CQ 

< 

< 

D 

oo 

oo 

Os 

Os 

m 

m 

m 

m 

oo 

oo 

fN 

fN 

«3 

cn 

15 

c 

o 

00 

2 

-o 

•5 

G 

O 

H 

«3 

cl 

CL 

CS 


L* 

rn 

u 

u 


x> 

.2 

13 

tC 


H 

uu 

Z 

* 

O 


u 

v 

'& 

CL 


g 

•*-» 

2 

w 

2 

C/3 

II 


$ 

C 


cd 

<D 

a3 

* 


w 

cd 

c* 


£ 


ON 

(N 


occupation  correlations  are  in  bold. 


Table  10.  Mean  Convergent  and  Divergent  Correlations  for  Ability  and  Skill  Level  Ratings 


Abilities 

Skills 

Mean 

Convergent  Correlations 

(same  occupation,  different  rater  types) 

SME-Project  Analysts 

All  8  occupations 

.56 

.73 

.64 

Officers 

.53 

.74 

.63 

Enlisted 

.60 

.71 

.65 

SME-0*NET  Analysts 

All  8  occupations 

.41 

.67 

.54 

Officers 

.36 

.67 

.52 

Enlisted 

.45 

.68 

.56 

Project  Analysts-0*NET  Analysts 

All  8  occupations 

.79 

.92 

.85 

Officers 

.77 

.92 

.84 

Enlisted 

.81 

.91 

.86 

Mean 

All  8  occupations 

.58 

.77 

.68 

Officers 

.55 

.78 

.66 

Enlisted 

.62 

.77 

.69 

Divergent  Correlations 

(different  occupations,  same  rater  type) 

SMEs 

All  8  occupations 

.44 

.50 

.47 

Officers 

.44 

.68 

.56 

Enlisted 

.58 

.75 

.66 

Project  Analysts 

All  8  occupations 

.48 

.59 

.54 

Officers 

.67 

.82 

.74 

Enlisted 

.40 

.63 

.51 

0*NET  Analysts 

All  8  occupations 

.60 

.65 

.63 

Officers 

.80 

.87 

.83 

Enlisted 

.61 

.75 

.68 

Mean 

All  8  occupations 

.51 

.58 

.54 

Officers 

.64 

.79 

.71 

Enlisted 

.53 

.71 

.62 

Note.  Mean  divergent  correlations  for  all  8  occupations  were  combined  across  28  between-occupation  correlations. 
Mean  divergent  correlations  for  officer  and  for  enlisted  are  each  based  on  six  correlations. 
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better  infer  the  abilities  and  skills  needed  to  perform  them.14  It  is  also  possible  that  SMEs 
and  project  analysts  were  factoring  in  Army-wide  (AW)  tasks.  Recall  that  0*NET  and 
project  analysts  were  instructed  to  make  their  ratings  based  on  occupation-specific 
information.  Even  so,  most  project  analysts  would  have  been  familiar  with  AW  tasks  and 
may  have  taken  that  into  account. 

•  The  0*NET  and  project  analysts  differentiated  enlisted  occupations  better  than  officer 
occupations.  For  example,  for  abilities,  the  average  project  analyst  divergent  correlation 
for  enlisted  occupations  (.40)  was  considerably  lower  than  that  for  officer  occupations 
(.67).  0*NET  analyst  data  yielded  average  correlations  of  .61  for  enlisted  and  .80  for 
officer  occupations.  Another  hypothesis/explanation  is  that  officer  occupations  are  truly 
less  divergent  (in  terms  of  required  skills  and  abilities)  and  this  is  being  accurately  picked 
up  by  the  analysts;  i.e.,  the  officer  occupations  may  not  provide  that  strong  a  test  of  the 
divergence  hypothesis.  Officer  jobs  require  the  performance  of  a  more  common  series  of 
functions  irrespective  of  job-specific  content  (e.g.,  planning  missions,  supervising  or 
leading  others,  managing  operations). 

•  Finally,  overall,  there  was  better  differentiation  on  average  for  the  abilities  than  for  the 
skills. 

Abi/ity  and S/c///L  eve/ Divergence  and  Con  vergence  Summary 

Probably  the  most  salient  finding  is  that  ability  level  ratings  appear  to  show  convergence 
and  divergence  for  enlisted  occupations.  The  mean  enlisted  ability  convergent  correlation  was 
.62  compared  to  the  average  divergent  correlation  of  .53.  In  our  sample  of  occupations,  there  was 
less  convergence  and  less  divergence  for  officer  occupations. 

For  skills,  convergence/divergence  was  better  when  all  8  occupations  were  included  in 
the  means  (.77  for  convergence  and  .58  for  divergence).  When  officers  and  enlisted  are 
compared  separately,  there  was  less  discrimination  on  skills.  We  suspect  that  this  is  because  skill 
level  means  were  fairly  low  for  enlisted  occupations  and  higher  for  officer  occupations  (see 
Table  14).  As  a  result,  including  the  cross-rank  correlations  increased  the  variance  for  the  across- 
all-eight  occupation  computations. 

Convergence  and  Divergence  of /PCs  and  GfVAs 

For  WCs  and  GW  As,  we  had  no  rater  types  to  compare.  We  defined  convergence  as  the 
multi-rater  reliability  (where  individual  raters  are  the  methods)  and  divergence  as  the  cross¬ 
occupation  correlations.  Work  context  occupation  intercorrelations  and  reliabilities  appear  in 
Table  1 1 .  Those  data  for  GW  As  appear  in  Table  12. 


14  Project  analyst  ratings  appeared  to  show  slightly  more  differentiation  despite  the  fact  that  we  would  have  expected 
their  mean  ratings  to  be  more  highly  correlated  due  to  systematic  rater  effects  (i.e.,  all  project  analysts  rated  all 
occupations).  Rater  effects  should  play  a  smaller  role  in  0*NET  analyst  ratings  since  two  different  groups  of  eight 
raters  each  rated  four  of  the  occupations. 
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Table  11.  Work  Context  Correlations  by  Occupation 


31A 

31B 

88A 

88M 

19A 

19K 

25A 

25U 

31 A 

.89 

31B 

.74 

.90 

88A 

.79 

.63 

.73 

88M 

.54 

.61 

.50 

.84 

19A 

.49 

.49 

.67 

.49 

.84 

19K 

-.05 

.20 

-.02 

.59 

.39 

.91 

25A 

.76 

.53 

.66 

.54 

.33 

-.08 

.89 

25U 

.53 

.58 

.57 

.81 

.42 

.46 

.67 

.88 

Note.  Correlations  are  based  on  the  SME  Work  Context  means  for  each  occupation.  Multi-rater  ICCs 
appear  on  the  diagonal. 


Table  12.  GWA  Level  Correlations  by  Occupation 


31A 

31B 

88A 

88M 

19A 

19K 

25A 

25U 

31A 

92 

31B 

.31 

.94 

88A 

.77 

.37 

.73 

88M 

.21 

.72 

.38 

.87 

19A 

.75 

.44 

.64 

.51 

.80 

19K 

.08 

.73 

.28 

.84 

.51 

.90 

25A 

.23 

-.21 

.14 

-.25 

-.03 

-.40 

.70 

25U 

.13 

.67 

.33 

.76 

.42 

.79 

-.04 

.88 

Note.  Correlations  are  based  on  the  SME  GWA  means  for  each  occupation.  Multi-rater  ICCs  appear  on  the  diagonal. 


Table  13  provides  a  summary  across  the  WC  and  GWA  correlation  matrices.  The  multi¬ 
rater  reliabilities  for  both  WCs  and  GWAs  are  reasonably  high,  indicating  convergence  within 
the  occupations,  and  are  considerably  higher  than  the  correlations  across  different  occupations 
(the  off-diagonal  elements  of  Tables  1 1  and  12),  suggesting  divergence.  The  GWAs  appear  to 
show  better  differentiation  for  officer  than  for  enlisted  occupations,  while  the  WCs  yield  a  little 
better  differentiation  for  enlisted  than  officer  occupations. 


Table  13.  Mean  Convergent  and  Divergent  Correlations  for  WC  and  GWA  Level  Ratings 

WC  GWA 


Convergent  Correlations 
(multi-rater  ICCs) 


All  8  occupations 

.86 

.84 

Officers 

.84 

.79 

Enlisted 

.88 

.90 

Divergent  Correlations 

(different  occupations) 

All  8  occupations 

.49 

.36 

Officers 

.62 

.42 

Enlisted 

.54 

.75 

Note.  Mean  divergent  correlations  for  all  8  occupations  were  combined  across  28  between-occupation  correlations. 
Mean  divergent  correlations  for  officer  and  for  enlisted  are  each  based  on  six  correlations. 
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How  Well  Do  SME  and  Analyst  Ratings  Agree? 

Table  10  showed  that  the  two  types  of  analyst  ratings  correlated  more  highly  with  each 
other  than  they  did  with  SMEs.  The  project  analyst  to  SME  correlation  was  higher  than  the 
0*NET  analyst  to  SME  correlation  for  both  abilities  and  skills. 

To  further  investigate  SME-analyst  differences,  we  examined  descriptive  statistics  and 
effect  sizes.  A  summary  of  the  descriptive  statistics  appears  in  Table  14.  The  findings  across 
scales  were  highly  consistent.  Regarding  mean  ratings,  SMEs  gave  the  highest  mean  ratings, 
regardless  of  the  scale,  project  analysts  gave  the  next  highest,  and  0*NET  analysts  gave  the 
lowest.  This  is  consistent  with  previous  0*NET  research  showing  that  incumbents  tend  to  give 
higher  ratings  than  analysts  (Peterson  et  al.,  1999).  In  looking  at  the  SDs  and  the  standard  errors 
of  the  mean  (SEm),  which  were  also  higher  for  SMEs,  it  is  important  to  note  that  some  of  the 
SME  groups  had  very  small  ns  making  these  statistics  more  difficult  to  compare  directly  to  those 
for  analysts. 

Another  salient  finding  in  Table  14  is  that  ability  importance,  ability  level,  skill 
importance,  and  skill  level  are  consistently  rated  lower  for  enlisted  than  officer  occupations, 
regardless  of  rater  type.  This  is  particularly  notable  for  the  skill  level  ratings  where  there  is 
consistently  one  SD  or  more  difference  between  the  officer  and  enlisted  skill  level  means.  In  at 
least  one  of  the  officer  workshops,  officers  commented  that  they  felt  the  skills  were  particularly 
relevant  to  officer  occupations  and  that  they  expected  their  enlisted  counterparts  would  score  low 
on  skills.  This  is  likely  because  the  skills  include  several  descriptors  that  get  at  management  and 
supervision. 

To  get  a  better  feel  for  the  magnitude  and  direction  of  differences  in  mean  ratings,  we 
computed  effect  sizes  for  rater  type  difference  on  each  ability  and  averaged  them  across  abilities. 
The  differences  are  not  trivial.  As  shown  in  Table  15,  the  standardized  mean  difference  ( d) 
between  SME  and  0*NET  analyst  ratings  across  all  abilities  was  1.21;  between  SME  and  project 
analyst  ratings  this  value  was  .76,  and  between  project  and  0*NET  analysts  it  was  .48.  The 
biggest  effect  sizes  between  0*NET  analysts  and  SMEs  were  typically  for  physical, 
psychomotor,  and  sensory  abilities,  with  SME  ratings  being  much  higher  than  0*NET  analyst 
ratings.  In  terms  of  specific  abilities,  the  largest  differences  were  for  Dynamic  Flexibility,  Night 
Vision,  Sound  Localization,  Reaction  Time,  Trunk  Strength,  Number  Facility,  and  Wrist-Finger 
Speed. 


Table  15  also  compares  the  magnitude  of  the  SME-analyst  differences  to  incumbent- 
analyst  differences  reported  during  0*NET  prototype  development  work  (Peterson  et  al.,  1999). 
In  that  work  D2 ,  the  squared  difference  between  rater  type  means,  was  used  to  measure  the 
difference.  For  abilities,  the  average  D2  =  .99  in  the  0*NET  prototype  work.  For  the  current 
effort,  the  D2  values  were  much  higher,  2.22  for  SME-0*NET  analyst  differences  and  1.33  for 
SME-project  analyst  differences.  It  should  be  noted,  however,  that  the  0*NET  prototype  data 
were  collected  for  32  occupations,  thus  the  occupations  sampled  could  have  played  a  role  in  the 
differences  between  the  two  studies. 
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Table  14.  Summary  Descriptive  Statistics  on  Ability  and  Skill  Scales  Across  Occupations  by 
Rater  Type 


Army  SME  Project  Analyst  0*NET  Analyst 


Mea 

Mdn 

Mdn 

Mean 

Mdn 

Mdn 

Mean 

Mdn 

Mdn 

n  of 

of 

of 

of  A/s 

of  SDs 

of 

of  A/s 

of  SDs 

of 

A/s 

SDs 

sem 

sem 

'SEm 

Ability  Importance 


All  Occupations 

3.53 

.88 

.33 

3.08 

.70 

.25 

2.92 

.59 

.21 

Officers 

3.73 

.82 

.33 

3.19 

.69 

.25 

3.06 

.58 

.21 

Enlisted 

3.34 

.95 

.33 

2.97 

.71 

.25 

2.78 

.59 

.21 

Ability  Level 

All  Occupations 

4.10 

1.15 

.43 

3.34 

.86 

.31 

2.96 

.78 

.27 

Officers 

4.35 

1.02 

.41 

3.55 

.89 

.32 

3.15 

.77 

.27 

Enlisted 

3.84 

1.28 

.44 

3.14 

.84 

.30 

2.78 

.78 

.28 

Skill  Importance 

All  Occupations 

3.13 

.78 

.29 

2.90 

.63 

.22 

2.73 

.55 

.20 

Officers 

3.80 

.66 

.27 

3.28 

.67 

.24 

3.04 

.58 

.21 

Enlisted 

2.45 

.90 

.32 

2.53 

.58 

.21 

2.42 

.53 

.19 

Skill  Level 

All  Occupations 

2.96 

.99 

.37 

2.73 

.84 

.30 

2.55 

.77 

.27 

Officers 

3.90 

.90 

.36 

3.28 

.79 

.28 

2.97 

.74 

.27 

Enlisted 

2.02 

1.09 

.38 

2.18 

.89 

.32 

2.13 

.79 

.28 

Note.  There  were  eight  raters  in  both  the  project  and  0*NET  groups,  but  the  number  of  Army  SMEs  varied  across 

occupations.  Sample  sizes  were  as  follows:  9,  9,  3,  9,  4,  6,  10,  9  for  31  A,  3 IB,  88A,  88M,  19A,  19K,  25 A,  and  25U, 

respectively. 
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Table  15.  Effect  Sizes  for  Rater  Types  on  Ability  Level  Scales 


Ability 

SME- 

Project 

Analyst 

M(d) 

Project 

SME-  Analyst- 

0*NET  0*NET 
Analyst  Analyst 

Project 

Analyst- 

0*NET 

Analyst 

MfD2) 

SME-  SME-  Analyst- 

Project  0*NET  Incumbent 

Analyst  Analyst  0*NET 

1 

Oral  Comprehension 

.51 

.87 

.44 

,ii 

.24 

.28 

.75 

2 

Written  Comprehension 

1.18 

1.40 

.13 

.08 

1.14 

1.14 

.94 

3 

Oral  Expression 

-.03 

.58 

.83 

.21 

.11 

.15 

.90 

4 

Written  Expression 

.35 

.56 

.25 

.18 

.46 

.33 

.79 

5 

Fluency  of  Ideas 

-.19 

.01 

.22 

.33 

.44 

.11 

.43 

6 

Originality 

.13 

.27 

.17 

.08 

.34 

.36 

1.39 

7 

Problem  Sensitivity 

.46 

.60 

.18 

.13 

.43 

.29 

1.83 

8 

Deductive  Reasoning 

.47 

.47 

.07 

.03 

.17 

.20 

.85 

9 

Inductive  Reasoning 

-.07 

-.12 

-.17 

.11 

.47 

.33 

.97 

10 

Information  Ordering 

.06 

-.03 

-.10 

.04 

.30 

.17 

.56 

11 

Category  Flexibility 

.45 

.25 

-.32 

.28 

.68 

.60 

1.18 

12 

Mathematical  Reasoning 

1.23 

.85 

-.55 

.44 

2.56 

1.01 

.76 

13 

Number  Facility 

1.29 

2.16 

.51 

.48 

1  0 

2.65 

4.27 

.83 

14 

Memorization 

.78 

1.81 

1.11 

6 

1  0 

.75 

3.01 

.67 

15 

Speed  of  Closure 

-.13 

.69 

1.11 

3 

.51 

.67 

.62 

16 

Flexibility  of  Closure 

.41 

.76 

.43 

.45 

.35 

1.02 

.81 

17 

Perceptual  Speed 

.24 

.51 

.32 

.23 

.39 

.67 

.71 

18 

Spatial  Orientation 

.60 

1.15 

.76 

.81 

.92 

2.18 

.53 

19 

Visualization 

.80 

1.23 

.44 

.49 

.85 

1.67 

.67 

20 

Selective  Attention 

.49 

1.50 

.98 

.78 

.46 

1.60 

2.15 

21 

Time  Sharing 

.57 

1.11 

.61 

.43 

.74 

1.58 

2.25 

22 

Arm-Hand  Steadiness 

.63 

1.17 

.49 

.16 

.94 

1.38 

.91 

23 

Manual  Dexterity 

1.07 

1.50 

-.12 

.37 

1.83 

1.94 

1.26 

24 

Finger  Dexterity 

1.68 

1.06 

-.44 

.29 

2.18 

1.08 

.92 

25 

Control  Precision 

.31 

.52 

.26 

.20 

.25 

.55 

.62 

26 

Multilimb  Coordination 

.27 

.38 

.20 

.10 

.26 

.37 

.86 

27 

Response  Orientation 

.71 

1.18 

.61 

.22 

1.5 

1.08 

1.49 

.69 

28 

Rate  Control 

.69 

1.48 

1.01 

6 

1.09 

3.67 

1.15 

2.6 

29 

Reaction  Time 

.70 

2.29 

1.43 

1 

1.65 

4.86 

1.13 

30 

Wrist-Finger  Speed 

1.56 

2.13 

.41 

.39 

4.91 

7.25 

.87 

Speed  of  Limb 

31 

Movement 

1.01 

1.97 

.97 

.81 

1.90 

4.70 

1.57 

32 

Static  Strength 

.97 

1.58 

.31 

.29 

1.66 

2.34 

.79 

33 

Explosive  Strength 

1.32 

1.86 

.43 

.78 

3.63 

5.65 

.63 

34 

Dynamic  Strength 

1.37 

2.13 

.21 

.30 

2.61 

3.72 

.76 

35 

Trunk  Strength 

1.88 

2.21 

.10 

.05 

3.06 

3.38 

.48 

36 

Stamina 

1.37 

1.98 

.45 

.28 

2.01 

3.26 

1.04 

37 

Extent  Flexibility 

1.04 

1.30 

.12 

.23 

1.86 

1.86 

.74 

38 

Dynamic  Flexibility 

2.28 

3.30 

.83 

.51 

6.93 

10.44 

1.07 

39 

Gross  Body  Coordination 

1.42 

1.71 

.13 

.11 

1.39 

1.72 

1.36 

35 


Table  15.  Effect  Sizes  for  Rater  Types  on  Ability  Level  Scales 


M(d) 

M(D2) 

Project 

Project 

SME- 

SME- 

Analyst- 

Analyst- 

SME- 

SME- 

Analyst- 

Project 

0*NET 

0*NET 

0*NET 

Project 

0*NET 

Incumbent 

Ability 

Analyst 

Analyst 

Analyst 

Analyst 

Analyst 

Analyst 

0*NET 

40 

Gross  Body  Equilibrium 

1.37 

1.72 

.07 

.13 

2.24 

2.30 

1.06 

41 

Near  Vision 

.82 

.87 

-.03 

.21 

1.19 

.63 

.73 

42 

Far  Vision 

.82 

1.43 

.62 

.56 

.72 

2.24 

.53 

43 

Visual  Color  Discrimin. 

1.06 

.97 

-.02 

.17 

1.55 

1.48 

.83 

4.0 

44 

Night  Vision 

.67 

2.37 

1.56 

9 

1  7 

1.23 

7.65 

.69 

45 

Peripheral  Vision 

.64 

1.34 

.82 

4 

1.50 

4.02 

.91 

46 

Depth  Perception 

.75 

1.16 

.45 

.34 

1.22 

1.81 

.50 

47 

Glare  Sensitivity 

.08 

.83 

.67 

.65 

.79 

1.12 

1.14 

48 

Hearing  Sensitivity 

.36 

.77 

.38 

.15 

.34 

.64 

.80 

49 

Auditory  Attention 

.66 

1.47 

.78 

.75 

1.02 

2.64 

1.54 

2.7 

50 

Sound  Localization 

.95 

2.30 

1.13 

0 

1.85 

8.34 

1.82 

51 

Speech  Recognition 

.09 

.24 

.18 

.25 

.69 

.48 

1.68 

52 

Speech  Clarity 

.53 

.84 

.25 

.17 

.50 

.77 

1.75 

Mean 

.74 

1.21 

.42 

.56 

1.33 

2.22 

.99 

Mean  \abs\ 

.76 

1.21 

.48 

Note  M(d)  is  the  mean  of  the  standardized  mean  differences  among  raters  across  occupations.  D:  is  the  squared 

difference  between  rater  type  means.  Data  in  the  Analyst-Incumbent  0*NET  column  are  from  the  0*NET  prototype 
development  work  (Peterson  et  al.,  1999). 


Table  16  reports  the  same  results  for  skills.  As  shown,  the  d  values  were  much  smaller  for 
skills  than  for  abilities.  On  average,  \d\  was  .28  for  project  analyst-0*NET  analyst,  .47  for  SME- 
project  analyst,  and  .64  for  SME-0*NET  analyst.  Also  the  magnitude  of  the  SME-0*NET 
analyst  difference  using  the  D2  value  (m  =  1 .23)  was  closer  to  the  value  achieved  for  analysts  and 
incumbents  in  the  0*NET  prototype  study  ( m  =  .95)  than  was  the  case  for  abilities. 
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Table  16.  Effect  Sizes  for  Rater  Types  on  Skill  Levd  Scales^ _ 

M(d)  M(D:) 


Skill 

Project 

SME-  SME-  Analyst- 

Project  0*NET  0*NET 

Analyst  Analyst  Analyst 

Project 

Analyst-  SME-  SME-  Analyst- 

0*NET  Project  0*NET  Incumbent 

Analyst  Analyst  Analyst  0*NET 

1 

Reading  Comprehension 

.87 

.89 

-.03 

.21 

.56 

.44 

.75 

1.0 

2 

Active  Listening 

.47 

3 

.68 

.32 

.70 

.81 

.94 

3 

Writing 

-.02 

.20 

.30 

.10 

.40 

.26 

.90 

4 

Speaking 

-.11 

.35 

.59 

.21 

.36 

.30 

.79 

1.0 

1.5 

2.2 

2.7 

5 

Mathematics 

4 

0 

.17 

.58 

3 

5 

.43 

1.1 

1.3 

2.5 

3.3 

6 

Science 

9 

6 

.40 

.23 

0 

5 

1.39 

7 

Critical  Thinking 

-.29 

.04 

.40 

.10 

.50 

.53 

1.83 

8 

Active  Learning 

-.03 

.26 

.35 

.13 

.24 

.37 

1.4 

.85 

9 

Learning  Strategies 

.75 

.86 

.21 

.16 

.89 

0 

.97 

1.2 

1.0 

10 

Monitoring 

-.54 

-.46 

.19 

.23 

3 

6 

.56 

11 

Social  Perceptiveness 

.22 

.28 

.04 

.10 

.40 

.39 

1.18 

12 

Coordination 

-.09 

.01 

.18 

.23 

.69 

.29 

.76 

13 

Persuasion 

.21 

.03 

-.31 

.41 

.70 

.99 

.83 

1.1 

1.0 

14 

Negotiation 

-.22 

-.13 

-.04 

.28 

2 

0 

.67 

15 

Instructing 

.32 

.27 

.03 

.15 

.72 

1.2 

.56 

.62 

16 

Service  Orientation 

.14 

.26 

.12 

.19 

3 

.86 

.81 

1.2 

1.2 

17 

Complex  Problem  Solving 

-.50 

-.48 

.03 

.05 

3 

8 

.71 

1.0 

2.8 

1.2 

3.5 

6.8 

18 

Operations  Analysis 

9 

2 

.63 

8 

5 

5 

.53 

1.0 

1.2 

3.1 

2.7 

19 

Technology  Design 

9 

4 

.19 

.73 

8 

8 

.67 

1.4 

2.2 

20 

Equipment  Selection 

.75 

.99 

.27 

.22 

6 

2 

2.15 

1.1 

2.0 

1.6 

2.9 

21 

Installation 

2 

9 

.99 

.62 

1 

5 

2.25 

22 

Programming 

-.11 

-.50 

-.45 

.41 

.37 

.38 

.91 

1.9 

1.4 

23 

Quality  Control  Analysis 

-.99 

-.73 

.50 

.47 

2 

4 

1.26 

24 

Operations  Monitoring 

.15 

-.14 

-.42 

.36 

.27 

.21 

.92 

25 

Operation  and  Control 

.16 

-.03 

-.17 

.13 

.80 

.53 

.62 

1.3 

1.6 

2.7 

26 

Equipment  Maintenance 

.65 

5 

.64 

.54 

3 

0 

.86 

27 

Troubleshooting 

.22 

.57 

.34 

.18 

.84 

.56 

.69 

28 

Repairing 

.25 

.47 

.02 

.21 

.55 

.91 

1.15 

29 

Systems  Analysis 

-.29 

-.30 

.03 

.14 

.18 

.46 

1.13 

30 

Systems  Evaluation 

-.29 

-.34 

-.12 

.16 

.56 

.58 

.87 

31 

Judg.  and  Dec.  Making 

-.28 

-.33 

-.01 

.13 

.76 

.88 

1.57 
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Table  16.  Effect  Sizes  for  Rater  Types  on  Skill  Level  Scales 


Skill 

M(d) 

MfD2) 

SME- 

Project 

Analyst 

SME- 

0*NET 

Analyst 

Project 

Analyst- 

0*NET 

Analyst 

Project 

Analyst- 

0*NET 

Analyst 

SME- 

Project 

Analyst 

SME- 

0*NET 

Analyst 

Analyst- 

Incumbent 

0*NET 

32 

Time  Management 

.68 

.43 

-.24 

.19 

.53 

.32 

.79 

1.8 

1.8 

33 

M.  of  Financial  Resources 

.87 

.99 

.07 

.14 

8 

1 

.63 

34 

M.  of  Material  Resources 

-.15 

.36 

.45 

.29 

.50 

.39 

.76 

35 

M.  of  Personnel  Resources 

-.36 

-.46 

-.07 

.14 

.22 

.56 

.48 

1.0 

1.2 

Mean 

.23 

.42 

.17 

.29 

4 

3 

.95 

Mean  \abs\ 

.47 

.64 

.28 

- m  . - — 

Note.  M(d)  is  the  mean  of  the  standardized  mean  differences  among  raters  across  occupations.  D"  is  the  squared 
difference  between  rater  type  means.  Data  in  the  Analyst-Incumbent  0*NET  column  are  from  the  0*NET  prototype 
development  work  (Peterson  et  al.,  1999). 


Are  Ability  Ratings  for  Army  Occupations  Similar  in  Quality  to  Ratings  for  Civilian 

Occupations?15 

We  compared  our  data  to  data  available  for  0*NET  abilities  to  determine  whether  the 
Army  data  were  comparable  in  quality  to  0*NET  data,  based  on  indicators  used  by  0*NET 
researchers.  This  section  focuses  only  on  abilities  because  comparable  0*NET  data  for  skills, 
work  context,  and  GWAs  were  not  available. 

A  bi/ity  F/ags 

0*NET  researchers  use  three  criteria  to  flag  the  ability  data  (Willison,  Byrum  & 
Tsacoumis,  2007). 

1 .  Not  Relevant.  The  level  rating  of  an  ability  is  flagged  as  “not  relevant”  for  a  particular 
occupation  if  two  or  fewer  of  the  eight  analysts  rated  its  importance  as  two  or  greater. 

2.  Large  SEm  Importance.  The  importance  rating  is  flagged  if  it  had  a  standard  error  of  the 
mean  {SEm)  greater  than  .51.  An  SEm  greater  than  .51  means  that  the  upper  and  lower 
bounds  of  the  confidence  interval  are  more  than  one  scale  point  away  from  the  observed 
mean. 

3.  Large  SEm  Level.  The  level  rating  is  flagged  if  it  had  a  standard  error  of  the  mean  {SEm) 
greater  than  .51,  for  the  same  reason  noted  in  the  previous  point. 

The  flags  are  used  by  0*NET  researchers  for  reporting  purposes.  For  example,  if  an 
ability  is  flagged  as  “not  relevant,”  the  0*NET  database  will  indicate  “not  relevant”  and  will  not 
provide  numeric  data  for  the  ability.  Similarly,  the  SEm  flags  determine  whether  importance  and 
level  information  will  be  reported. 


15  0*NET  data  for  this  section  was  only  available  for  abilities. 
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Table  1 7  shows  the  result  of  applying  the  ability  flags  to  the  project  and  0*NET  analyst 
data  for  Army  occupations,  along  with  0*NET  results  from  six  cycles  of  data  collection.16  As 
shown,  there  were  far  fewer  “Not  Relevant”  flags  for  Army  occupations  than  are  typical  for 
civilian  occupations  in  the  Standard  Occupational  Classification  {SOC) — the  system  used  by 
the  U.S.  government  to  categorize  all  occupations.  Some  physical  (e.g.,  explosive  strength, 
dynamic  flexibility)  and  sensory  (e.g.,  night  vision,  sound  localization)  descriptors  were  flagged 
as  not  relevant  for  over  half  of  the  SOC  occupations,  but  were  relevant  for  all  of  the  Army 
occupations  in  our  sample.  Similarly,  psychomotor  descriptors  (e.g.,  response  orientation,  speed 
of  limb  movement)  were  flagged  as  not  relevant  for  over  30%  of  the  SOC  occupations,  but  they 
were  relevant  for  the  Army  occupations.  These  results  seem  to  make  sense.  A  smaller  proportion 
of  civilian  than  military  occupations  require  physical,  psychomotor,  and  sensory  abilities.  These 
results  suggest  that  military  occupations  require  a  wider  range  of  abilities  than  do  civilian 
occupations. 

The  SEm  flags  were  established  to  capture  those  ratings  deemed  to  have  insufficient 
agreement  across  raters.  As  shown  in  Table  17,  SOC  importance  ratings  on  abilities  are  rarely 
flagged  on  SEm,  and  none  of  the  Army  occupation  importance  ratings  were  flagged.  The  flags  on 
ability  level  were  different.  Here  the  0*NET  analyst  data  for  Army  occupations  yielded  fewer 
flags  than  the  project  analyst  data  did.  0*NET  analyst  level  flags  for  Army  occupations  were  in 
a  comparable  range  of  those  observed  for  0*NET  analyst  level  ratings  for  SOC  occupations. 

Descriptive  Statistics  and  Se/iabi/ities for  A bi/ities 

We  compared  ratings  on  abilities  made  by  0*NET  analysts  for  this  project  to  those  made 
by  0*NET  analysts  in  the  most  recently  published  cycle  of  analyst  results  (cycle  7;  Willison, 
Byrum,  &  Tsacoumis,  2007).  Cycle  7  had  101  occupations.  As  shown  in  Table  18,  the  mean  of 
the  mean  importance  rating  on  Army  occupations,  2.92,  was  higher  than  the  means  of  the  means 
across  101  occupations.  Even  so,  the  individual  means  were  within  the  range  observed  for  SOC 
occupations.  The  SD  and  SEm  for  importance  were  also  within  the  range  observed  for  the 
0*NET  occupations.  The  level  scale  ratings  are  on  the  high  side  compared  to  those  in  the 
0*NET  cycle  7  data.  Even  so,  the  SD  and  SEM  values  are  comparable  to  those  observed  for 
0*NET  cycle  7  occupations. . 

The  data  in  Table  1 9  show  that  0*NET  analysts  rated  Army  occupations  just  as  reliably 
as  they  rated  civilian  occupations,  at  least  for  this  set  of  eight  Army  occupations.  The  average 
and  range  of  the  reliabilities  were  very  similar  across  the  Army  and  cycle  7  occupations. 

Do  the  Score  Profiles  for  Army  Occupations  Look  Like  Those  for  Similar  Civilian 

Occupations? 


16  Cycle  1  data  were  from  the  0*NET  prototype  and  were  not  analyzed  in  the  same  way  as  other  cycles.  Cycle  8  and 
9  data  are  not  yet  available.  We  did  not  include  Army  SME  data  in  this  analysis  because  the  statistics  computed  in 
this  section  are  dependent  upon  sample  size  for  interpretation  and  the  number  of  Army  SMEs  varied  across 
occupations,  making  the  data  difficult  to  compare  to  the  other  rater  types  that  all  had  8  raters. 
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We  used  the  multi-trait  multi-method  approach  (Campbell  &  Fiske,  1959)  again,  this  time 
to  assess  convergence  and  divergence  of  Army  and  Standard  Occupational  Classification  ( SOC) 
data.  For  these  analyses,  Army  and  SOC  are  treated  as  different  methods  of  measuring  the  same 
occupation.  Using  the  crosswalk  in  0*NET,  we  identified  civilian  occupations  in  the  SOC  that 
appeared  to  be  the  best  possible  match  with  our  target  Army  occupations.  The  result  appears  in 
Table  20.  For  all  of  the  occupations  except  25A  and  25U,  the  0*NET  crosswalk  provided  an 
obvious  direct  match.  For  25A  and  25U,  we  examined  several  descriptions  and  chose  the  one 
that  appeared  to  be  the  best  match.  While  the  0*NET  crosswalk  identified  counterparts  for  the 
19A  and  19K  jobs,  it  did  not  provide  any  data  on  those  counterpart  occupations. 

Next,  for  each  descriptor  set,  we  correlated  vectors  of  mean  ratings  for  the  Army  and 
SOC  occupations.  For  abilities  data,  we  used  0*NET  analyst  ratings  because  0*NET  reports 
analyst  data.  For  all  of  the  other  descriptor  sets  (skills,  GW  As,  and  work  context)  we  used  SME 
data  because  0*NET  data  on  these  descriptors  are  from  incumbents.  The  results  for  correlations 
of  the  ability  level  profiles  appear  in  Table  21.  As  shown,  Army  occupations  were  fairly  highly 
correlated  with  their  civilian  counterparts  in  terms  of  ability  level  ratings.  For  example,  the  3 1 A 
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Table  18.  Comparison  of  Civilian  and  Army  Descriptive  Statistics  for 
Ability  Ratings  made  by  0*NET  Analysts 


Occupation 

Ability  Importance 

Ability  Level 

Mean  of 
Ms 

Mdn  of 
SDs 

Mdn  of 
SEmS 

Mean  of 
Ms 

Mdn  of 
SDs 

Mdn  of 
SEms 

31A 

3.01 

.53 

.19 

3.10 

.71 

.25 

31B 

3.00 

.67 

.24 

3.00 

.73 

.26 

88A 

3.09 

.73 

.26 

3.15 

.86 

.30 

88M 

2.78 

.64 

.23 

2.84 

.74 

.26 

19A 

3.20 

.53 

.19 

3.35 

.76 

.27 

19K 

2.86 

.53 

.19 

2.80 

.76 

.27 

25A 

2.94 

.53 

.19 

3.00 

.74 

.26 

25U 

2.49 

.52 

.18 

2.47 

.90 

.32 

Mean 

2.92 

.59 

.21 

2.96 

.78 

.27 

101  Occupations  (0*NET  Cycle  7) 

Mean 

2.59 

.60 

.21 

2.33 

.74 

.26 

SD 

.26 

.11 

.04 

.35 

.10 

.04 

Min 

2.00 

.35 

.13 

1.58 

.46 

.16 

Max 

3.18 

.89 

.31 

3.02 

.99 

.35 

Table  19.  Comparability  of  Civilian  and  Army  Reliabilities  for  Ability  Ratings  made  by 
0*NET  Analysts  


Ability  Importance  Ability  Level 


0*NET  Cycle  7 

Army  Occupations 

0*NET  Cycle  7 

Army  Occupations 

#  Occupations 

101 

8 

101 

8 

Mean 

.92 

.92 

.92 

.91 

Min 

.84 

.88 

.84 

.88 

Max 

.99 

.96 

.99 

.95 

Note  All  reliabilities  are  ICC  (3,8)  estimates  for  0*NET  analysts. 


Table  20.  Standard  Occupational  Classification  (SOC)  Counterparts  for  Army  Target 
Occupations  _ 

Army  Target  Occupation  _ Standard  Occupational  Classification  ( SOC )  Counterpart 

3 1 A  Military  Police  (Officer)  33-1012.00  Is1  Line  Supervisors/Managers  of  Police  and 

Detectives 

3 IB  Military  Police  (Enlisted)  33-3051.01  Police  Patrol  Officer 

88 A  Transportation  (Officer)  1 1  -307 1 .00  Transportation  Manager 

88M  Motor  Transport  Operator  (Enlisted)  53-3032.00  Truck  Driver,  Heavy  and  Tractor-Trailer 

19A  Armor  (Officer)  55-1013.00  Armored  Assault  Vehicle  Officers* 

19K  Ml  Armor  Crewman  (Enlisted)  55-3013.00  Armored  Assault  Vehicle  Crew  Members* 

25 A  Signal  (Officer)  11-3021 .00  Computer  and  Information  Systems  Managers 

25U  Signal  Support  Systems  Specialist  (Enlisted)  15-1071.00  Network  and  Computer  Systems  Administrators 

*These  SOC  codes  were  listed  as  counterpart  civilian  occupations  by  0*NET.  Even  so,  0*NET  does  not  contain 

data  for  them. 
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Table  21.  Correlations  of  Ability  Level  Profile  Means  for  Target  Army  and  SOC  Occupations 

_ _ SOC  Occupations  _ Army  MOS 

Network 


m 


ability  level  means  correlated  .80  with  the  ability  level  means  for  1st  Line  Supervisors  of  Police 
and  Detectives.  On  average,  the  Army  occupations  correlated  more  highly  with  their  counterparts 
(r  =  .73)  than  they  did  with  the  non-counterparts  (r  =  .41)  or  with  the  other  Army  occupations  (r 
=  .60). 


Table  22  summarizes  the  results  across  abilities,  skills,  WCs,  and  GW  As.  The  results 
were  very  consistent.  Army  occupations  correlated  most  highly  with  their  SOC  counterparts  for 
all  descriptor  sets;  they  did  not  correlate  as  highly  with  non-counterpart  SOC  occupations.  Both 
convergence  and  divergence  were  strong  for  the  ability  level  ratings  in  particular.  It  is  worth 
noting,  though,  that  the  correlations  among  occupations  based  on  the  GWA  ratings  were 
considerably  lower  than  those  for  the  other  three  descriptor  sets. 


Table  22.  Summary  of  Intercorrelations  Among  Army  and  Selected  SOC  Counterpart 
Occupations 


Ability 

Level 

Skill 

Level 

GWA 

Level 

Work  Context 

Mean 

Convergent  Correlation — The  average 
correlation  between  SOC  and  Army  counterparts 

.73 

.64 

.43 

.66 

.61 

Divergent  Correlation — The  average  correlation 
between  SOC  and  Army  non-counterparts 

.41 

.47 

.17 

.40 

.  .36 

The  average  correlation  among  SOCs  only 

.37 

.51 

.42 

.69 

.50 

The  average  correlation  among  Army  only 

.60 

.50 

.36 

.49 

.49 

Note.  Correlations  for  abilities  are  based  on  0*NET  analyst  ratings.  Correlations  for  skills,  GW  As,  and 

work  context  are  based  on  SME  ratings. 


Summary 

In  this  chapter  we  made  comparisons  between  intraclass  correlation  coefficients  (ICCs), 
mean  descriptor  profiles,  and  effect  sizes,  and  used  the  multi-trait  multi-method  approach  to 
address  five  key  evaluation  questions: 

•  Are  Army  occupations  rated  reliably  using  0*NET  rating  scales? 

•  Do  ratings  on  0*NET  rating  scales  differentiate  Army  occupations? 

•  How  well  do  SME  and  analyst  ratings  of  Army  jobs  agree? 

•  Are  ratings  on  Army  occupations  similar  in  quality  to  ratings  on  civilian  occupations? 

•  Are  Army  occupation  profiles  similar  to  those  for  their  civilian  counterparts? 

Do  raters  rate  Arm y  occupations  reliably  using  O  *NE T  rating  scales  l 

•  For  abilities  (Table  4),  including  eight  raters  as  recommended  by  0*NET  procedures,  both 
project  and  0*NET  analyst  data  yielded  averaged  reliabilities  of  about  .90,  well  above  the 
recommended  .80.  However,  SME  reliabilities  were  generally  lower  than  recommended. 
The  .80  level  was  reached  for  two  and  three  occupations,  respectively,  for  level  and 
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importance.  In  general,  six  to  25  SME  raters  would  be  required  to  achieve  the 
recommended  reliability  level  of  .80. 17 

Reliability  estimates  were  generally  higher  for  skills  than  abilities  for  all  types  of  raters. 
On  skills  (Table  5),  Army  SMEs  achieved  the  recommended  .80  reliability  level  for  all 
occupations  but  one,  25A  Signal  officers,  which  also  yielded  anomalous  results  for  other 
descriptors  (see  Chapter  1  for  possible  explanations).  Project  and  0*NET  analyst  ratings 
achieved  reliabilities  of  about  .93. 


•  For  work  context  descriptors  (rated  only  by  SMEs),  on  average,  the  work  context  ratings 
reached  the  recommended  reliability  level  of  .80.  The  only  exception  was  88A 
Transportation  officer  which  only  had  5  raters.  Discussion  with  SMEs  indicated  that  there 
may  be  additional  contextual  factors  appropriate  for  the  Army  environment,  especially 
those  relevant  to  combat,  that  should  be  added. 

•  SME  ratings  on  GWAs,  on  average,  reached  the  recommended  level  of  reliability.  The 
only  occupation  which  legitimately  yielded  lower  than  this  recommended  level  was  25A, 
Signal  officer.  The  other  occupations  with  reliabilities  below  this  level  had  less  than  eight 
raters. 

•  On  the  whole,  project  analysts  and  0*NET  analysts  produced  more  than  adequate  levels  of 
reliability  for  abilities  and  skills.  SMEs  also  produced  adequate  levels  of  reliability  for 
work  context  and  GWAs. 

Do  ratings  on  O  *NET  rating  sca/es  ti///erentiate  Army  occupations? 

•  SMEs  better  differentiated  Army  occupations  than  did  project  analysts  or  0*NET  analysts; 
their  average  correlation  between  different  occupations  was  .47  across  abilities  and  skills. 
Project  analysts  differentiated  more  than  0*NET  analysts  (average  rs  =  .54  and  .63, 
respectively).  These  correlations  certainly  indicate  some  degree  of  differentiation  of 
occupations. 

•  The  ability  level  ratings  appear  to  show  both  convergence  and  divergence  for  enlisted 
occupations.  The  mean  enlisted  convergent  correlation  for  ability  level  was  .62,  as 
compared  to  the  average  divergent  correlation  of  .53.  In  our  sample  of  occupations,  there 
was  less  convergence  and  less  divergence  for  officer  occupations  based  on  ability. 

•  For  skills,  convergence/divergence  was  better  when  all  eight  occupations  were  included  in 
the  means  (.77  for  convergence  and  .58  for  divergence).  When  officers  and  enlisted  were 
analyzed  separately,  there  was  less  discrimination  on  skills.  Officer  skill  level  means  were 
an  SD  or  more  higher  than  those  for  enlisted  occupations.  Therefore,  there  is  considerably 


17  SMEs  received  only  basic  introduction  to  the  abilities  and  skills  ratings,  and  both  sets  of  analysts  received 
extensive  training. 
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more  variance  in  the  skill  level  ratings  when  both  officer  and  enlisted  occupations  are 
included. 

•  The  multi-rater  reliabilities  for  both  WCs  and  GWAs  are  reasonably  high,  indicating 
convergence  within  the  occupations,  and  they  are  considerably  higher  than  the  correlations 
across  different  occupations,  suggesting  divergence. 

/fow  we// do  SA/E  and  ana/yst  ratings  agree ? 

•  Analyst  ratings,  on  average,  correlated  more  highly  with  each  other  ir  =  .85)  than  they  did 
with  SME  ratings  (r  =  .64  and  .54,  respectively  for  project  and  0*NET  analysts).  For 
abilities,  the  project  analyst-to-SME  correlation  was  higher  than  the  0*NET  analyst-to- 
SME  correlation.  For  skills,  both  types  of  analyst  ratings  correlated  about  the  same  with 
SME  ratings. 

•  SMEs  rated  importance  and  level  on  abilities  and  skills  higher  than  did  either  project  or 
0*NET  analysts;  project  analysts  gave  slightly  higher  ratings  than  did  0*NET  analysts. 

•  Effect  sizes  showed  these  differences  to  be  fairly  large  for  SMEs,  as  compared  to  the  two 
sets  of  analysts.  The  standardized  mean  difference  ( d)  between  SME  and  0*NET  analyst 
ratings  across  all  abilities  was  1.21;  the  mean  d  between  SME  and  project  analyst  ratings 
was  .74,  and  the  mean  d  between  project  and  0*NET  analysts  was  .42.  Effect  sizes  for 
skills  were  much  smaller,  about  half  as  large. 

•  It  appears  that  project  and  0*NET  analysts  reach  a  fairly  high  level  of  agreement,  both  in 
terms  of  correlations  between  ratings  and  magnitude  of  ratings.  On  the  other  hand,  SMEs 
reach  only  moderate  levels  of  agreement  with  either  set  of  analysts,  although  agreement  is 
somewhat  better  for  skills  than  abilities. 

Are  ratings  on  Arm y  occupations  simi/ar  in  qua/ity  to  ratings  on  civi/ian  occupations ? 

•  There  were  far  fewer  “Not  Relevant”  flags  for  Army  occupations  than  is  typical  for  SOC 
occupations;  this  is  almost  certainly  due  to  the  greater  prevalence  of  requirements  for 
physical,  psychomotor,  and  sensory  abilities  in  Army  occupations  as  compared  to  SOC 
occupations. 

•  The  SEm  flags  were  comparable  to  those  found  for  SOC  occupations,  indicating  similar 
agreement  among  raters. 

Are  Arm y  occupation  pro/i/es  simi/ar  to  t /lose /or  their  c/vi/ian  counterparts ? 

•  Ability,  skill,  GW  A,  and  WC  ratings  for  Army  occupations,  on  average,  correlated  higher 
with  ability,  skill,  GWA,  and  WC  ratings  for  their  SOC  counterparts  (mean  r  =  .61)  and 
did  not  correlate  as  highly  ratings  for  non-counterpart  SOC  occupations  (mean  r  =  .36). 
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IV.  Analysis  of  Fit  Between  Major  Duties  and  Generalized  Work  Activities 

Purpose 

Generalized  Work  Activities  in  the  0*NET  are  defined  as  “an  aggregation  of  similar 
occupation  activities/behaviors  that  underlie  the  accomplishment  of  major  work  functions”  (p. 
106,  Peterson,  et  al.,  1999).  The  Army’s  Major  Duties  (MDs)  are  defined  as  higher-order 
statements  that  describe  a  definable  and  nontrivial  duty,  or  responsibility,  for  which  a  Soldier  (or 
Officer)  is  accountable  to  perform,  which  has  stakeholders  to  whom  the  results  (i.e.,  outputs)  are 
important  and  meaningful,  and  which  entails  work  of  significant  complexity  and  duration. 

As  part  of  the  evaluation  of  the  utility  of  the  0*NET  for  Army  use,  the  evaluation  of  the 
relationship  between  these  two  sets  of  descriptors  seemed  in  order.  If  0*NET  GWAs  appear  to 
be  related  to  or  “cover”  all  or  many  of  the  Army  MDs,  then  they  would  seem  to  be  quite  useful 
for  providing  a  higher-order  description  of  Army  occupations  (i.e.,  one  that  would  encompass  the 
major  duties  performed  in  the  Army). 

To  make  this  evaluation,  we  asked  non-incumbent  raters  to  rate  the  degree  to  which  each 
MD  was  “covered”  by  each  GWA,  with  no  occupation  specifically  identified  (i.e.,  the  MD  might 
be  imagined  to  be  a  part  of  any  of  several  occupations;  no  particular  occupation  was  implicated). 
The  rating  of  “coverage”  of  duties  by  GWAs  was  an  experiment  to  identify  the  degree  to  which 
project  staff  considered  the  GWAs  to  “cover”  or,  in  the  words  of  the  0*NET  definition  of  a 
GWA,  “underlie”  the  major  duties.  Raters  were  instructed  to  read  the  definition  and  anchors  for 
the  first  GWA  and  then  make  a  judgment  regarding  the  extent  to  which  the  GWA  covers  the  MD 
using  the  following  scale: 

0  =  little  or  no  coverage  of  MD  (most  or  all  of  MD  not  subsumed  by  the  GWA) 

1  =  partial  coverage  of  MD  (some  of  MD  subsumed  by  the  GWA) 

2  =  strong  coverage  of  MD  (most  of  MD  subsumed  by  the  GWA) 

Although  it  is  difficult  to  exactly  specify  what  might  be  considered  a  sufficient  amount  of 
“coverage”  of  a  major  duty  by  GWAs,  we  believe  it  can  be  plausibly  argued  that  if  at  least  one 
GWA  shows  “ strong "  coverage  of  a  major  duty,  that  is  sufficient — given  the  rating  instructions 
that  we  used  in  this  experiment.  Likewise,  if  no  GWAs  show  at  least  " partial ”  coverage  for  an 
MD,  then  coverage  is  insufficient.  We  recognize  that  the  degrees  of  coverage  between  these  two 
extreme  kinds  of  scores  are  more  difficult  to  characterize  in  terms  of  sufficiency,  and  that 
observers  can  reasonably  differ  on  this  matter. 

MD  x  GWA  Exercise  Method 

Practically  speaking,  the  rating  task  had  to  be  divided  up.  There  are  41  0*NET  GWAs 
and  98  major  duties  (as  described  in  Chapter  II).  This  yields  over  4,000  separate  ratings. 
Consequently,  we  undertook  a  preliminary  set  of  judgments  to  divide  the  major  duties  into  three 
groups  corresponding  to  the  three  major  factors  of  the  GWAs  (identified  in  early  research  with 
the  0*NET;  Peterson  et  al,  1999,  pp.  122):  (a)  Working  with  and  Directing  Others,  (b)  Working 
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with  Information,  and  (c)  Manual,  Physical  or  Technical  Activities.  Three  judges  (the  project 
director  and  two  senior  consultants  acting  as  technical  advisors  for  the  project)  independently 
categorized  each  MD  into  one  of  these  three  categories.  Where  all  three  judges  agreed,  the  MD 
was  put  into  that  category.  Where  there  was  not  universal  agreement,  the  differences  were 
discussed  and  a  consensus  was  reached.  As  a  result,  17  major  duties  were  placed  into  two 
categories,  and  two  MDs  were  placed  in  three  categories  because  they  appeared  complex  enough 
to  require  judgments  about  the  “coverage”  by  GWAs  in  more  than  one  category.  Appendix  B 
shows  the  placement  of  the  major  duties  into  the  three  categories.  Note  that  this  process  assumes 
that  the  GWAs  that  were  not  rated  against  an  MD  would  have  been  rated  “0”  (little  or  no 
coverage)  for  the  MD.  We  believe  this  is  a  safe  assumption;  we  tried  to  err  on  the  side  of  over¬ 
including  the  MD  by  putting  it  in  multiple  categories  if  we  questioned  its  categorization. 

The  three  judges  determined  that  a  small  subset  of  the  MDs  were  non-technical 
performance  categories,  derived  mainly  from  performance  rating  categories  used  in  prior  Army 
projects.  In  this  way,  they  were  fundamentally  different  from  other  MDs  (higher-order 
activities/tasks)  and  from  the  GWAs.  By  removing  these  MDs  from  the  exercise,  the  judges 
essentially  determined  that  they  were  not  covered  by  the  GWAs,  because  they  are  different  in 
nature  from  them  (i.e.,  GWAs  are  aggregations  of  activities).  They  might  map  better  against 
other  descriptor  sets  in  0*NET,  such  as  Work  Styles.  Those  MDs  were: 

•  Demonstrate  Military  Presence 

•  Manage  Own  Duties  and  Responsibilities 

•  Demonstrate  Extra  Effort  and  Personal  Initiative  on  the  Occupation 

•  Manage  Own  Professional  Development 

•  Demonstrate  Personal  Integrity 

•  Exhibit  Self-Control 

•  Model  Correct  Behavior  to  Soldiers 

We  created  four  forms  in  Excel  workbooks  for  the  exercise.  Many  MDs  had  been 
categorized  into  the  Working  with  Information  factor;  therefore,  we  split  those  MDs  across  two 
rating  forms.  The  other  two  rating  forms  were  for  the  Working  With  and  Directing  Others  factor 
and  the  Manual,  Physical  and  Technical  Activities  factor.  MDs  were  listed  in  the  rows;  GWAs 
were  listed  in  the  columns.  Full  definitions  for  both  the  MDs  and  GWAs  were  also  provided  in 
workbooks  and  appear  in  Appendix  C  of  this  report.  Raters  were  instructed  to  study  the  GWAs 
and  their  definitions,  then  read  the  definition  of  the  first  MD  and  rate  the  extent  to  which  that 
MD  was  covered  by  the  GWA  using  the  three-category  rating  scale  described  above. 

Raters  were  project  staff,  comprised  of  a  combination  of  HumRRO  and  ARI  personnel. 
Each  rater  was  assigned  to  complete  2  to  4  forms,  depending  upon  his  or  her  availability.  Ratings 
were  made  by  a  total  of  1 1  staff:  seven  with  Ph.Ds,  one  highly  experienced  master’s  level  staff 
member,  and  three  junior  staff  with  relevant  experience  and  education.  Ten  raters  completed  the 
Manual  Physical  and  Technical  Activities  form;  eight  raters  completed  each  of  the  other  three 
forms. 
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Results 


fte/iabi/ity 

Reliability  of  the  ratings  completed  by  the  judges  was  assessed  by  computing  the 
correlations  between  each  pair  of  rater’s  ratings,  across  the  entire  set  of  ratings  within  each  of  the 
four  exercises;  computing  the  mean  of  these  correlations;  and  then  correcting  the  mean 
correlation  to  the  number  of  raters  in  the  exercise.  This  method  corresponds  closely  to  the  R<d 
and  R(k)  cases  of  the  intraclass  correlation  (McGraw  &  Wong,  1996). 

The  reliability  estimates  for  the  ratings  are  shown  in  Table  23.  Note  that  the  single-rater 
reliability  should  be  used  to  compare  the  reliabilities  across  rating  exercises  because  different 
exercises  had  different  numbers  of  raters.  The  “Manual,  Physical,  and  Technical  Activities” 
exercise  showed  the  highest  reliability,  while  the  two  “Working  With  Information”  exercises 
showed  the  lowest  reliabilities.  All  of  the  ratings  are  at  acceptable  levels;  none  of  the  single-rater 
reliabilities  are  less  than  .42  and  the  lowest  k-rater  reliability,  of  most  concern  here,  was  .85. 


Table  23.  Reliability  estimates  for  GWA  Extent-of-Coverage  Ratings 


Number 

Number  of 

Number 

of 

0*NET 

of 

Number 

Major 

Generalized 

Ratings 

of 

One-rater 

k-rater 

Type  of  Activities 

Duties 

Work  Activities 

Made 

Raters 

Reliability 

Reliability 

Manual,  Physical  and  Technical 

42 

8 

336 

10 

.60 

.94 

Working  With  and  Directing  Others 

29 

17 

493 

8 

.55 

.91 

Working  With  Information  (A)* 

19 

23 

437 

8 

.42 

.85 

Working  With  Information  (B)* 

18 

23 

414 

8 

.42 

.85 

rater’s  profile  of  ratings,  and  then  computing  the  average  correlation  between  raters  (the  "number  of  ratings  made"  is 
the  number  of  observations  in  the  "profile,"  over  which  the  correlation  was  computed).  The  k-rater  reliability  was 
computed  by  applying  the  Spearman-Brown  Prophecy  Formula  to  the  single-rater  reliability. 

♦The  "Working  With  Information"  Major  Duties  were  split  into  two  rating  tasks,  but  the  GW  As  remained  constant 
for  the  two  tasks. 


Descr/p/ive  Statis/ics 

We  computed  the  mean  and  standard  deviation  of  the  ratings  of  each  GWA  for  each  MD. 
The  marginal  means  and  standard  deviations  of  the  matrix  of  mean  ratings  were  computed;  that 
is,  the  average  of  each  row  (an  MD)  of  means  and  the  average  of  each  column  (a  GWA)  of 
means  were  computed,  as  well  as  their  standard  deviations.  Detailed  summaries  of  these  statistics 
for  each  of  the  four  rating  exercises  are  shown  in  the  tables  in  Appendix  D. 

We  identified  two  main  indicators  of  the  extent  to  which  an  MD  is  covered  by  GWAs — 
(a)  the  maximum  rating  and  (b)  the  distribution  of  mean  ratings  across  GWAs  for  a  MD.  The 
maximum  mean  rating  for  each  MD  across  all  the  GWAs  (for  which  it  was  rated)  would  identify 
MDs  that  are  mostly  subsumed  by  one  or  more  GWAs.  The  second  indicator  combined  the 
mean  ratings  (i.e.,  the  strength  of  coverage)  and  the  number  of  GWAs  receiving  that  mean  rating. 
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We  prepared  a  frequency  distribution  showing  the  number  of  mean  GWA  ratings  for  each  MD 
that  fell  into  the  following  intervals: 

•  <  .50  (meaning  that,  on  average,  more  than  half  the  judges  rated  the  GWA  as  having 
“little  or  no  coverage”) 

•  .50  to  .99  (meaning  that,  on  average,  at  least  half  the  judges  rated  the  GWA  as  having 
“partial  coverage”) 

•  1 .0  to  1 .50  (meaning  that,  on  average,  all  the  judges  rated  the  GWA  as  having  “partial 
coverage”) 

•  >1.50  (meaning  that,  on  average,  more  than  half  the  judges  rated  the  GWA  as  having 
“strong  coverage”) 

Table  24  shows  the  results  in  order  by  the  number  of  GWAs  rated,  on  average,  greater 
than  .99  (A  comparable  table  ordered  by  item  number  appears  in  Appendix  E).  The  first  two 
columns  of  the  table  show  the  MD  ID#  and  title.  The  next  four  columns  show  the  distribution  of 
the  mean  ratings  across  the  four  intervals  described  above.  The  next  column  shows  the  number 
of  mean  ratings  that  were  greater  than  .99.  Note  that  the  total  number  of  ratings  varies  across  the 
major  duties,  for  reasons  described  above  (i.e.,  the  number  of  ratings  for  a  MD  varied  across 
forms  and  some  MDs  were  repeated  across  forms).  The  final  column  shows  our  judgments  about 
the  coverage  of  each  MD  by  the  0*NET  GWAs.  Those  judgments  were  made  according  to  the 
following  decision  rules: 

•  Full  Coverage — For  the  MD,  at  least  one  GWA  had  a  mean  rating  greater  than  1 .50 

•  High  Partial  Coverage — For  the  MD,  no  GWAs  had  a  mean  rating  greater  than  1.50, 
but  at  least  six  GWAs  had  mean  ratings  greater  than  .99 

•  Low  Partial  Coverage — For  the  MD,  no  GWAs  had  a  mean  rating  greater  than  1 .50, 
and  three  to  five  GWAs  had  mean  ratings  greater  than  .99 

•  Not  Covered — For  the  MD,  no  GWAs  had  a  mean  rating  greater  than  1 .50,  and  fewer 
than  three  GWAs  had  ratings  greater  than  .99 

The  first  rule  was  intended  to  account  for  the  situation  where  a  specific  MD  is  specific  to 
one  GWA  and  well-covered  by  it.  The  GWAs  tend  to  be  broader  than  the  MDs,  therefore,  an  MD 
can  fall  almost  completely  within  one  GWA.  For  example,  MD  #67,  Maintain  Physical  Fitness, 
is  characterized  as  “Full”  coverage,  but  has  only  one  GWA  rated  greater  than  zero — but  that 
GWA  is  Performing  General  Physical  Activities  (defined  as  “Performing  physical  activities  that 
require  considerable  use  of  your  arms  and  legs  and  moving  your  whole  body...”).  Similarly, 

MDs  #  42  and  #  43,  Drive  Wheeled  Vehicles  and  Drive  Water  Craft,  respectively,  have  just  one 
GWA  rated  greater  than  zero — but  it  is  #  21,  Operating  Vehicles,  Mechanized  Devices,  or 
Equipment  (defined  as  “Running,  maneuvering,  navigating,  or  driving  vehicles  or  mechanized 
equipment  such  as  forklifts,  passenger  vehicles,  aircraft,  or  water  craft”).  These  GWAs  do  seem 
to  fully  cover  the  MDs  for  which  they  were  so  highly  rated. 
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Table  24.  Summary  of  Ratings  of  "Coverage"  By  GWAs  of  Army  Major  Duties  (By  Number 
of  A  verage  Ratings  > .  99) 


Number  of  GWAs  with  Average 

Ratings: 


ID# 

Major  Duty 

<0.50 

.50  to 
.99 

1.00  to 
1.50 

>1.50 

Avg. 

Ratings 

>.99 

Judged 

Coverage 

83 

Plan  and  organize  operations/missions  and  team  tasks 

Provide  technical  guidance  and  advice  on  the  installation, 

17 

9 

9 

5 

14 

Full 

29 

maintenance,  and  use  of  equipment 

17 

13 

5 

5 

10 

Full 

81 

Contribute  to  team  planning 

17 

13 

7 

3 

10 

Full 

59 

Schedule  patients  and  medical  services 

5 

7 

2 

9 

Full 

40 

Monitor  and  control  financial  resources 

8 

7 

6 

2 

8 

Full 

62 

Collect  and  analyze  weather  and  environmental  data 
Troubleshoot  and  repair  personal  computers  and  computer 

9 

6 

5 

3 

8 

Full 

20 

networks 

18 

6 

6 

1 

7 

Full 

31 

Analyze  intelligence  data 

8 

9 

5 

1 

6 

Full 

45 

Order  supplies  and  equipment 

8 

9 

5 

1 

6 

Full 

75 

Direct  peers  and  individual  team  members 

Install  and  maintain  personal  computers  and  peripheral 

7 

4 

3 

3 

6 

Full 

18 

equipment 

20 

6 

4 

1 

5 

Full 

25 

Collect  and  decode  electronic  signals 

13 

5 

2 

3 

5 

Full 

26 

Analyze  electronic  signals 

14 

4 

4 

1 

5 

Full 

27 

Provide  data  processing  and  programming  support 

11 

7 

3 

2 

5 

Full 

33 

Collect  information  from  and  on  individuals  and  groups 

25 

10 

3 

2 

5 

Full 

37 

Prepare  and  process  forms 

15 

3 

3 

2 

5 

Full 

39 

Write  documents  and  correspondence 

15 

3 

4 

1 

5 

Full 

44 

Inspect,  store,  and  issue  supplies 

16 

10 

5 

0 

5 

High  Partial 

94 

Coordinate  with  other  units  and  non-Army  personnel 

21 

14 

3 

2 

5 

Full 

7 

Navigate  from  point  to  point 

21 

6 

4 

0 

4 

High  Partial 

19 

Install  and  maintain  computer  networks 

21 

6 

2 

2 

4 

Full 

38 

Maintain  records  and  files 

18 

1 

2 

2 

4 

Full 

58 

Provide  medical  treatment 

27 

17 

3 

1 

4 

Full 

61 

Perform  laboratory  procedures 

21 

6 

3 

1 

4 

Full 

76 

Support  peers  and  individual  team  members 

12 

1 

1 

3 

4 

Full 

84 

Direct  and  motivate  individual  Soldiers 

9 

4 

3 

1 

4 

Full 

87 

Administer  personnel  actions  and  procedures 

14 

5 

3 

1 

4 

Full 

88 

Manage  and  monitor  operations/missions  and  team  tasks 

10 

3 

3 

1 

4 

Full 

4 

Inspect  and  maintain  weapons 

3 

2 

1 

2 

3 

Full 

6 

Troubleshoot  and  repair  weapons 

4 

1 

1 

2 

3 

Full 

10 

Scout  and  identify  targets 

14 

6 

1 

2 

3 

Full 

11 

Fire  heavy  direct  fire  weapons 

4 

1 

2 

1 

3 

Full 

12 

Fire  indirect  fire  weapons 

20 

8 

2 

1 

3 

Full 

14 

Install  and  maintain  electronic  equipment 

3 

2 

2 

1 

3 

Full 

15 

Troubleshoot  and  repair  electronic  equipment 

4 

1 

1 

2 

3 

Full 

16 

Install  and  maintain  electrical  and  power  transmission  systems 
Troubleshoot  and  repair  electrical  and  power  transmission 

3 

2 

3 

0 

3 

Low  Partial 

17 

systems 

3 

2 

1 

2 

3 

Full 

22 

Operate  personal  computers  and  networks 

17 

3 

2 

1 

3 

Full 

24 

Send  and  receive  radio  messages 

17 

3 

3 

0 

3 

High  Partial 

35 

Troubleshoot  and  repair  mechanical  equipment 

3 

2 

1 

2 

3 

Full 

46 

Manage  and  control  traffic 

10 

10 

3 

0 

3 

High  Partial 

51 


Table  24.  Summary  of  Ratings  of  " Coverage  "  By  GW  As  of  Army  Major  Duties  (By  Number 
of  Average  Ratings  >  .99) 


Number  of  GWAs  with  Average 

Ratings: 


ID# 

Major  Duty 

<0.50 

.50  to 
.99 

1.00  to 
1.50 

>1.50 

Avg. 

Ratings 

>.99 

Judged 

Coverage 

47 

Operate  hand-operated  power  excavating  equipment 

4 

1 

2 

1 

3 

Full 

48 

Operate  heavy  equipment 

4 

1 

1 

2 

3 

Full 

50 

Repair  metal  structures  or  parts 

4 

1 

3 

0 

3 

Low  Partial 

51 

Construct  metal  or  steel  structures 

4 

1 

3 

0 

3 

Low  Partial 

60 

Provide  counseling  and  other  interpersonal  interventions 

10 

4 

2 

1 

3 

Full 

63 

Conduct  land  surveys 

17 

3 

2 

1 

3 

Full 

64 

Deliver  presentations 

31 

6 

1 

2 

3 

Full 

77 

Train  peers  and  individual  team  members 

11 

3 

2 

1 

3 

Full 

82 

Contribute  to  team  coordination 

9 

5 

1 

2 

3 

Full 

86 

Communicate  information  to  Soldiers,  peers,  and  superiors 

32 

5 

1 

2 

3 

Full 

89 

Direct  and  lead  platoons/squads/teams 

9 

5 

2 

1 

3 

Full 

92 

Build  and  manage  platoon/squad/team  cohesion 

10 

4 

1 

2 

3 

Full 

93 

Engage  and  negotiate  with  host  nationals  and  local  leaders. 

10 

4 

0 

3 

3 

Full 

2 

Handle  demolitions  or  mines 

2 

4 

1 

1 

2 

Full 

3 

Engage  in  hand-to-hand  combat 

6 

0 

1 

1 

2 

Full 

5 

Fire  direct  fire  weapons 

5 

1 

2 

0 

2 

Low  Partial 

9 

Provide  emergency  first  aid 

39 

7 

1 

1 

2 

Full 

23 

Record  and  document  audiovisual  information 

19 

2 

0 

2 

2 

Full 

30 

Translate  foreign  languages 

13 

8 

1 

1 

2 

Full 

32 

Control  individuals  and  crowds 

20 

3 

1 

1 

2 

Full 

34 

Inspect  and  maintain  mechanical  equipment 

2 

4 

0 

2 

2 

Full 

36 

Operate  gas  and  electric  powered  equipment 

5 

1 

1 

1 

2 

Full 

41 

Load  and  unload  supplies 

6 

0 

0 

2 

2 

Full 

49 

Install,  maintain,  and  repair  plastic  and  fiberglass 

4 

2 

2 

0 

2 

Low  Partial 

52 

Install,  maintain,  and  repair  pipe  assemblies 

4 

2 

2 

0 

2 

Low  Partial 

53 

Construct  wooden  buildings  and  structures 

5 

1 

1 

1 

2 

Full 

54 

Construct  masonry  buildings  and  structures 

5 

1 

1 

1 

2 

Full 

55 

Produce  technical  drawings  and  illustrations 

19 

2 

1 

1 

2 

Full 

57 

Prepare  patients  and  equipment  for  medical  procedures 

21 

2 

1 

1 

2 

Full 

65 

Reproduce  printed  materials 

6 

0 

2 

0 

2 

Not 

74 

Contribute  to  team  tasks 

13 

2 

1 

1 

2 

Full 

85 

Train  and  coach  Soldiers 

12 

3 

0 

2 

2 

Full 

91 

Support  individual  Soldiers 

12 

3 

0 

2 

2 

Full 

1 

Protect  against  NBC  hazards 

4 

3 

1 

0 

1 

Low  Partial 

8 

Maintain  personal  and  operational  security 

6 

1 

1 

0 

1 

Not 

13 

Drive  track  vehicles 

6 

1 

0 

1 

1 

Full 

28 

Produce  maps,  overlays,  or  range  cards 

12 

10 

1 

0 

1 

High  Partial 

42 

Drive  wheeled  vehicles 

7 

0 

0 

1 

1 

Full 

43 

Drive  water  craft 

7 

0 

0 

1 

1 

Full 

56 

Prepare  food  and  beverages 

5 

2 

1 

0 

1 

Low  Partial 

67 

Maintain  physical  fitness 

7 

0 

0 

1 

1 

Full 

78 

Help  peers  and  individual  team  members 

14 

2 

1 

0 

1 

Low  Partial 

79 

Monitor  peer  and  individual  team  member  performance 

12 

4 

1 

0 

1 

Low  Partial 

80 

Monitor  team  performance 

13 

3 

1 

0 

1 

Low  Partial 

52 


Table  24.  Summary  of  Ratings  of  " Coverage "  By  GW  As  of  Army  Major  Duties  (By  Number 
of  A  verage  Ratings  > .  99)  


Number  of  GWAs  with  Average 

Ratings: 

Avg. 

Ratings 

>.99 

ID# 

Major  Duty 

<0.50 

.50  to 
.99 

1.00  to 
1.50 

>1.50 

Judged 

Coverage 

21 

Operate  electronic  equipment 

6 

2 

0 

0 

0 

Not 

73 

Follow  orders  and  rules 

16 

l 

0 

0 

0 

Not 

Note.  Rating  scale  was;  0=little  or  no  coverage;  l=partial  coverage;  2=strong  coverage. 


Using  these  criteria,  67  of  the  87  MDs  had  “Full”  coverage,  five  had  “High  Partial” 
coverage,  1 1  had  “Low  Partial”  coverage,  and  four  were  not  covered. 

If  we  examine  the  15  MDs  in  the  “Not  Covered”  and  “Low  Partial  Coverage”  categories, 
we  perhaps  can  identify  some  kinds  of  duties  for  which  the  Army  might  need  supplemental 
GWAs.  We  have  made  an  initial  grouping  of  these  1 5  MDs  as  follows: 

Trades-Related: 

Install  and  maintain  electrical  and  power  transmission  systems 

Repair  metal  structures  or  parts 

Construct  metal  or  steel  structures 

Install,  maintain,  and  repair  plastic  and  fiberglass 

Install,  maintain,  and  repair  pipe  assemblies 

Hazard/Combat  Related: 

Fire  direct  fire  weapons 

Protect  against  NBC  hazards 

Maintain  personal  and  operational  security 

Team-Related: 

Help  peers  and  individual  team  members 

Monitor  peer  and  individual  team  member  performance 

Monitor  team  performance 

Miscellaneous: 

Operate  electronic  equipment 
Prepare  food  and  beverages 
Follow  orders  and  rules 

It  is  somewhat  puzzling  that  the  “Trades-related”  group  appears  here;  it  would  seem  that 
a  sizable  set  of  civilian  occupations  would  include  these  types  of  activities.  A  reviewer  suggested 
that  this  could  have  been  due  to  raters’  interpretations  of  the  GWAs  which  are  broad.  Possibly, 
but  we  tend  to  agree  with  the  raters.  The  means  and  standard  deviations  of  the  ratings  appear  in  a 
research  note  published  separately  from  this  document  (Russell  et  al.,  2008). 
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It  is  not  surprising  that  at  least  some,  combat-specific  MDs  received  low-partial  coverage 
ratings.  Of  the  thirteen  combat-specific  MDs,  nine  received  full  coverage  judgments  (e.g.,  handle 
demolitions  or  mines,  troubleshoot  or  repair  weapons),  one  received  a  high  partial  coverage 
judgment  (navigate  from  point-to-point  on  the  ground),  and  the  remaining  three  (fire  dirct 
weapons,  protect  against  NBC  hazards,  and  maintain  personal  and  operational  security)  were 
deemed  low  partial  or  not  covered.  This  might  seem  counterintuitive;  it  could  be  argued  that 
even  though  an  MD  is  linked  or  rated  as  being  covered  by  multiple  GWAs,  the  most  critical 
aspect  of  it  is  not  sufficiently  covered  by  any  of  the  linked  GWAs.  Our  data  do  not  support  that 
notion.  Given  the  instructions  that  we  gave  raters,  they  should  not  have  rated  an  MD  highly  if  a 
critical  aspect  of  it  was  not  covered.  It  is  important  to  remember  that  the  SOC  (0*NET)  does 
include  occupations  that  use  weapons  and  demolitions,  drive  heavy  equipment,  and  provide 
emergency  and  disaster  response.  Given  that,  it  makes  sense  that  only  a  few  combat  MDs  would 
not  be  covered  well. 

It  would  seem  that  one  or  more  Army-specific  GWAs  might  usefully  be  created  for  the 
Hazard/Combat  Related  grouping  and/or  for  the  Team-Related  grouping.  If  this  were 
undertaken,  then  other  MDs  having  similar  content  to  these  two  groupings  might  be  identified  to 
inform  the  development  of  such  GWAs. 

The  Miscellaneous  grouping  is  a  hodge-podge:  “Operate  electronic  equipment”  could  be 
its  own  GWA,  as  perhaps  could  “Prepare  food  and  beverages,”  while  “Follow  orders  and  rules” 
does  not  seem  like  an  activity  in  the  sense  of  the  other  MDs. 

Several  approaches  could  be  used  to  remedy  MD  areas  with  lack  of  GWA  coverage.  One 
would  be  to  add  several  new  specific  GWAs;  another  might  be  to  add  only  a  few  broad  GWAs; 
another  might  be  to  expand  the  definitions  of  existing  GWAs  to  include  such  MDs. 
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V.  Summary  and  Discussion 


The  evaluation  focused  primarily  on  the  usefulness  of  the  0*NET  system  for  Army 
occupation  analysis  for  selection  and  classification  purposes.  We  focused  on  the  appropriateness 
of  0*NET  descriptors  that  would  typically  be  used  in  an  Army  occupational  analysis  for 
selection  and  classification  purposes.  These  included  abilities,  skills,  GWAs,  and  WCs. 

Our  approach  was  designed  to  address  the  following  six  questions: 

1 .  Are  Army  occupations  rated  reliably  using  0*NET  rating  scales? 

2.  Do  ratings  on  0*NET  rating  scales  differentiate  Army  occupations? 

3.  How  well  do  SME  and  analyst  ratings  agree? 

4.  Are  ratings  on  Army  occupations  similar  in  quality  to  ratings  on  civilian  occupations? 

5.  Are  Army  occupational  profiles  similar  to  those  for  their  civilian  counterparts? 

6.  How  well  do  0*NET’s  work  requirements  descriptors,  particularly  GWAs,  cover 
Army  job  requirements? 

Addressing  the  first  five  questions  required  collection  of  ratings  on  the  selected  0*NET 
descriptors — abilities,  skills,  GWAs  and  WCs — for  target  Army  occupations.  Our  approach  was 
to  produce  data  for  the  military  occupations  that  could  be  compared  to  civilian  0*NET  data. 
Therefore,  it  was  important  to  follow  processes  currently  used  by  0*NET  for  data  collection.  For 
0*NET,  job  incumbents  provide  ratings  on  occupational  tasks,  skills,  GWAs,  and  work  context 
areas.  Ability  ratings  for  occupations  are  collected  from  trained  analysts.  Analysts  have  also  been 
trained  to  rate  skills,  although  those  data  have  been  used  only  experimentally  until  recently. 
Accordingly,  we  collected  skill,  GWA,  and  work  context  data  from  SMEs  for  each  target  Army 
occupation.  We  collected  analyst  ratings  on  the  abilities  and  skills  for  target  Army  occupations, 
following  the  procedures  used  in  collecting  0*NET  analyst  ratings  for  SOC  occupations 
(Donsbach  et  al.,  2003).  In  the  0*NET  analyst  training,  analysts  rate  occupations  using  a 
standard  set  of  descriptive  materials — tasks  for  the  occupation  and  incumbents’  ratings  of  those 
tasks,  GWAs,  and  work  context  descriptors.  To  use  a  parallel  approach  in  the  evaluation,  we 
developed  task  lists  for  the  target  Army  occupations.  In  sum,  we  collected  task,  skill,  GWA,  and 
work  context  ratings  from  SMEs  and  ability  ratings  from  analysts.  To  allow  comparison  of  SME 
and  analyst  data,  we  also  asked  SMEs  to  rate  abilities  and  analysts  to  rate  skills. 

The  sixth  question  required  somewhat  different  data.  If  0*NET  GWAs  appeared  to  be 
related  to  or  “cover”  all  or  many  of  the  Army  job  requirements  that  cut  across  occupations,  here 
called  Major  Duties  (MDs),  then  they  would  seem  to  be  quite  useful  for  providing  a  higher-order 
description  of  Army  occupations  (i.e.,  one  that  would  encompass  the  major  duties  performed  in 
the  Army).  If  not,  additions  or  revisions  to  the  GWAs  might  be  needed.  In  preparation  for  this 
assessment  of  coverage,  we  developed  a  list  of  MDs  based  on  prior  research  and  obtained 
feedback  on  it  from  Army  SMEs.  To  make  this  evaluation,  we  asked  non-incumbent  raters  to 
rate  the  degree  to  which  each  MD  was  “covered”  by  each  GWA,  with  no  occupation  specifically 
identified  (i.e.,  the  MD  might  be  imagined  to  be  a  part  of  any  of  several  occupations;  no 
particular  occupation  was  implicated).  Findings  for  the  six  questions  were  discussed  at  the  end  of 
Chapters  III  and  IV. 
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In  this  chapter  we  summarize  results  and  make  recommendations  regarding  each 
descriptor  set.  These  recommendations  have  to  be  tempered  with  some  caveats.  First,  four  officer 
and  four  enlisted  occupations  -  out  of  hundreds  of  Army  occupations  -were  included  in  the 
research;  clearly  that  limits  the  strength  of  our  conclusions.  Also,  our  research  design  (by 
necessity)  did  not  allow  us  to  fully  cross  raters  and  occupations.  (This  would  have  required 
SMEs  to  rate  occupations  other  than  their  own.)  Thus,  we  cannot  fully  assess  rater  effects.  Also, 
we  limited  the  descriptors  we  considered  to  ones  likely  to  be  useful  for  selection  and 
classification  purposes;  we  cannot  draw  conclusions  about  0*NET’s  other  descriptor  sets  such  as 
Work  Styles.  Finally,  because  we  focused  on  selection  and  classification,  we  cannot  draw 
conclusions  about  the  usefulness  of  0*NET  for  other  purposes  such  as  training  needs 
assessment. 


Abilities 


Summary  of  A  bi/ity  Results 

The  0*NET  abilities  were  designed  to  cover  the  full  range  of  individual  differences  in 
ability  that  have  been  studied  over  last  century,  and  they  were  derived  from  both  military  and 
civilian  studies  (Fleishman  et  al.,  1999).  While  one  could  argue  about  different  taxonomic 
systems  (e.g.,  Fleishman’s  physical  abilities  as  opposed  to  the  one  developed  by  Hogan  (1991),  it 
would  be  very  difficult  to  argue  that  the  abilities  do  not  capture  the  range  of  replicable,  stable 
individual  differences  reported  in  the  literature. 

The  0*NET  analyst  training  package  appears  to  work  fairly  well.  Results  presented  in 
Chapter  III  suggest  that  trained  analysts  make  reliable  ability  ratings.  With  some  training,  SMEs 
might  do  so  as  well,  albeit  SMEs  are  likely  to  take  issue  with  the  lack  of  Army-specificity  in  the 
anchors.  We  also  found  that  trained  project  and  0*NET  analysts  were  able  to  rate  the  abilities 
with  about  the  same  level  of  agreement  and  reliability  observed  for  0*NET  analyst  ratings  of 
civilian  occupations. 

We  also  found  that  ability  level  ratings  show  appropriate  convergence  and  divergence  for 
enlisted  occupations.  That  is,  the  correlation  between  occupations  on  abilities  was  low  relative  to 
the  correlations  among  different  rater  types.  This  suggests  that  the  abilities  would  provide  a 
useful  basis  for  distinguishing  jobs  for  classification  purposes.  We  did,  however,  find  that 
project  analysts  drew  more  distinctions  than  0*NET  analysts.  This  suggests  that  trained  analysts 
should  be  ones  who  know  something  about  Army  jobs.  We  also  found  that  the  abilities  were  less 
distinguishing  for  officer  occupations,  but  it  is  possible  that  officer  occupations  are  truly  more 
similar  to  each  other  than  enlisted  ones  are,  since  officers  have  common  responsibilities 
regardless  of  occupation. 

Finally,  when  we  applied  the  multi-trait,  multi-method  approach  to  assess  the 
convergence/divergence  of  Army  and  SOC  data,  we  found  that  Army  occupations  correlated 
most  highly  with  their  SOC  counterparts  (mean  r  =  .73);  they  did  not  correlate  as  highly  with 
non-counterpart  occupations  (mean  r  =  .41). 
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Abi/ity  Descriptor  Decommendations 


•  Use  0*NET ability  questionnaires  “as  is  "  or  close  to  it.  In  all,  the  0*NET  abilities 
yield  reliable  ratings  that  show  both  convergent  and  divergent  validity.  Retaining  the 
level  anchors  and  descriptors  “as  is”  would  allow  direct  comparison  with  civilian 
occupations  in  the  SOC. 

•  Consider  developing  a  pool  of  Army  0*NET  analysts  to  make  ability  ratings.  Based 
on  our  data,  trained  analysts  who  have  some  experience  with  Army  jobs  are  likely  to 
provide  the  most  useful  judgments.  The  process  0*NET  uses  for  training  raters  and 
obtaining  data  appears  to  work  well.  Furthermore,  the  process  is  efficient:  trained 
analysts  can  rate  the  abilities  and  skills  for  an  occupation  in  less  than  two  hours  and 
can  be  asked  to  rate  large  numbers  of  occupations  in  a  reasonably  short  amount  of 
time.  If  SMEs  were  to  make  ability  ratings  (a)  they  would  need  training  and  (b) 
additional  Army-specific  anchors  might  need  to  be  added  to  the  level  scales.  Given 
that  SME  time  is  very  precious  and  at  times  hard  to  obtain,  using  trained  analysts 
would  appear  to  be  the  better  alternative. 

•  Link  the  0*NET  ability  taxonomy  to  predictor  variables  of  interest.  One  useful 
addition  might  be  to  conduct  a  linkage  exercise  mapping  existing  and  experimental 
predictors  (e.g.,  ASVAB  subtests  and  other  measures,  such  as  the  Army’s 
experimental  psychomotor  tests)  to  the  0*NET  abilities  taxonomy.  This  might 
provide  a  route  to  investigating  occupation  clusters  based  on  those  predictors. 

Skills 


Summary  of. S/ci// Desu/ts 

All  rater  types  rated  skills  very  reliably,  and  there  were  no  large  differences  in  the 
magnitude  of  ratings  made  by  different  rater  types  on  skills.  When  all  eight  occupations  were 
considered,  the  skill  ratings  showed  differentiation  across  occupations.  That  differentiation 
diminished  when  we  looked  within  enlisted  and  officer  occupations,  most  of  the  differentiation 
probably  resulting  from  differences  between  officer  and  enlisted  occupation  skill  level 
requirements.  Skill  level  means  were  considerably  lower,  on  average,  for  entry-level  enlisted 
jobs  than  for  entry-level  officers;  it  makes  sense  that  higher  levels  of  skills  are  more  likely  to 
come  into  play  as  Soldiers  move  up  the  ranks.  As  with  the  abilities,  Army  occupations  were 
more  highly  correlated  with  counterpart  SOC  occupations  than  non-counterpart  occupations, 
another  piece  of  evidence  supporting  skill  rating  validity. 

Ski// Descriptor  D ecom  m  en  c/at  ions 

Our  recommendations  for  the  skill  descriptors  are  not  as  clear  cut  as  they  were  for  the 
abilities.  While  the  skill  ratings  were  of  high  quality,  they  did  not  well  discriminate  among  our 
small  sample  of  enlisted  occupations.  This  could  be  a  function  of  the  number  and  type  of 
occupations  included  in  the  current  research  Therefore,  one  recommendation  is  to  collect 
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0*NET  skill  data  for  additional  occupations.  It  might  also  be  useful  to  collect  the  skill  ratings  for 
SL1,  SL2,  and  SL3  to  see  how  well  skills  might  differentiate  level. 

Two  additional  avenues  warrant  further  exploration: 

•  Assess  the  utility  of  using  skill  level  data  to  guide  promotion  tool  development.  The 
skills  showed  promise  for  distinguishing  among  skill  levels.  The  Army  has  been 
seeking  efficient  ways  to  develop  promotion  measures  and  perhaps  skill  level 
information  could  form  a  basis  for  measure  development  (e.g.,  promotion  tools  would 
be  targeted  toward  levels  on  skill  scales  instead  of  creating  new  promotion 
instruments  for  each  occupation). 

•  Consider  potential  uses  of  skill  ratings  for  officer  development.  Several  of  the  skill 
descriptors  deal  with  managerial  duties.  It  could  be  useful  to  present  data  collected 
from  this  project  to  individuals  responsible  for  designing  the  Basic  Officer  Leader 
Course  (BOLC  I  and  II)  and  get  their  input  as  to  the  potential  usefulness  of  the  skill 
descriptors  for  the  officer  ranks. 

Work  Context 


Summary  of  fV or/c  Context  S esu/ts 

While  the  empirical  data  for  the  work  context  variables  were  strong  (e.g.,  SMEs  rated 
them  reliably,  they  differentiated  jobs),  verbal  reports  from  the  SME  workshops  suggested  areas 
in  which  the  work  context  descriptors  might  be  improved. 

tV or/c  Context  Descriptor  fiecomm  en  Sal ions 

One  consistent  theme  in  SME  workshops  had  to  do  with  large  differences  between  in¬ 
garrison  and  deployment  contexts.  Our  observation  is  that  this  had  its  greatest  effect  on  (a)  work- 
oriented  descriptors  (i.e.,  GWAs,  work  context,  and,  in  particular,  KWAs),  and  (b)  specific  rating 
scales  (e.g.,  frequency  scales  with  absolute  anchors).  Handling  this  issue  will  require  further 
study.  It  is  not  a  simple  matter  of  adding  a  deployed  versus  in-garrison  scale.  We  have  learned 
that  over  the  course  of  current  operations  in  Iraq  and  Afghanistan  that  work-related  requirements 
can  vary  considerably,  both  over  time  and  within  the  same  country  depending  on  duty  location. 
We  recommend  research  to  develop  an  efficient  set  of  deployment-context  descriptors. 

One  possibility  would  be  to  develop  a  set  of  5-10  descriptors  that  characterize  the  main 
features  of  the  mission  context.  Years  ago,  a  U.S.  Army  Special  Forces  (SF)  general  told  the  first 
author  of  this  report  that  there  were  basically  two  kinds  of  SF  work:  (a)  door  kicking  and  (b) 
well-digging — and  that  Soldiers  who  were  good  at  one  were  not  necessarily  good  at  the  other.  It 
may  be  that  at  the  broadest  level  there  are  two  dimensions  are  direct  action  and  diplomatic/aid. 
The  key  is  to  determine  what  descriptors  distinguish  these  contexts.  For  example,  the  level  of 
interaction/cooperation  with  foreign  nationals  or  indigenous  people  would  seem  to  be  one 
distinguishing  factor.  One  method  of  developing  mission  profiles  would  be  to  ask  Soldiers  to  rate 
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different  missions  using  the  0*NET  work  context  variables.  This  could  illuminate  what 
descriptors  distinguish  missions. 

To  be  useful,  such  research  would  need  to  concisely  and  accurately  define  a  very  limited 
number  of  such  descriptors,  and,  most  crucially,  demonstrate  that  they  have  a  moderating  effect 
on  the  ratings  made  on  the  other  sets  of  descriptors  thought  to  be  affected  by  these  larger 
contextual  factors  (i.e.,  the  GWAs,  KWAs,  and  the  other  context  descriptors).  Also  some  thought 
needs  to  go  into  how  these  descriptors,  once  defined,  would  be  used. 

Based  on  comments  from  SMEs  in  the  workshops,  possible  additions  to  the  descriptor  set 
include  the  following: 

•  Work  Pace.  One  WC  item  asks  how  important  is  it  to  keep  a  pace  set  by  machinery  or 
equipment.  It  prompted  a  discussion  about  how  work  pace  varies  for  combat  and  non¬ 
combat  activities.  During  battle,  the  pace  is  influenced  by  available  technology  as  well  as 
other  factors  (e.g.,  own  and  enemy  troop  strength). 

•  Working  with  little  sleep.  A  discussion  about  long  work  hours  that  Soldiers  keep  during 
deployments  also  prompted  a  discussion  about  sleep  deprivation.  SMEs  said  that  sleep 
deprivation  affects  the  mission  and  is  influenced  by  the  mission.  It  appears  that  junior 
officers,  in  particular,  often  operate  with  very  little  sleep.  It  would  seem  that  working 
with  little  sleep  could  be  an  important  context  element  to  capture. 

•  Communicating  with  indigenous  people,  host  nation  counterparts,  peers  and  superiors. 
Another  theme  that  tended  to  come  from  officers  had  to  do  with  communications.  During 
deployments,  junior  officers  have  been  assigned  to  work  with  host  nation  counterparts 
(e.g.,  Iraqi  junior  officers)  and  to  negotiate  with  indigenous  leaders  about  activities  and 
resources.  While  much  of  the  negotiation  takes  place  through  an  interpreter,  it  is  still  a 
challenge  to  deal  with  the  cultural  differences  in  the  negotiation.  The  junior  officer 
SMEs  also  thought  it  could  be  useful  to  add  WC  items  that  reflect  their  need  to  persuade 
or  influence  their  peers  or  superiors. 

•  Travel,  overseas  travel,  and  extended  time  away  from  home.  SMEs  mentioned  that  items 
could  deal  with  being  separated  from  one’s  normal  environment  (e.g.,  in  a  foreign 
country)  for  an  extended  period  of  time. 

•  Lifting  heavy  weight.  Some  SMEs  felt  that  lifting  heavy  weight  was  not  sufficiently 
covered  in  the  WC  items. 

SMEs  suggested  that  Army  examples  could  be  added  to  some  items  to  make  them 
clearer.  For  some  items  this  could  be  done  without  disturbing  the  content  of  the  item.  For 
example,  a  WC  item  about  protective  gear  could  add  examples  of  combat  gear  to  the  list  of 
civilian  gear  in  that  item.  This  kind  of  change  could  be  useful  if  SMEs  were  to  make  ratings  on 
the  work  context  scales  operationally. 
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GWAs 


Summary  of  GWA  R esu//s 

Our  data  suggest  that  if  the  Army  were  to  collect  GWA  ratings  from  about  1 5  to  1 8 
SMEs  for  each  occupation,  a  recommended  multi-rater  reliability  of  .80  would  be  achieved.  The 
data  also  suggest  that  those  GWA  ratings  would  differentiate  occupations  (Table  12  and  Table 
13).  The  convergent  correlations  between  Army  occupations  and  SOC  counterparts,  in 
conjunction  with  the  divergent  correlations  between  Army  and  non-counterpart  occupations, 
provide  additional  support  for  the  convergent  and  divergent  validity  of  the  GWA  ratings  made  by 
SMEs. 


The  task  of  assessing  the  coverage  of  MDs  by  GWAs  yielded  some  surprises.  Based  on 
the  criteria  we  imposed,  67  of  the  87  MDs  had  “Full”  coverage,  five  had  “High  Partial” 
coverage,  1 1  had  “Low  Partial”  coverage,  and  four  were  not  covered.  Interestingly,  the  1 5  with 
low  partial  or  no  coverage  were  not  necessarily  Army-specific.  We  grouped  them  as  follows: 

Trades-Related: 

Install  and  maintain  electrical  and  power  transmission  systems 

Repair  metal  structures  or  parts 

Construct  metal  or  steel  structures 

Install,  maintain,  and  repair  plastic  and  fiberglass 

Install,  maintain,  and  repair  pipe  assemblies 

Hazard/Combat  Related: 

Fire  direct  fire  weapons 

Protect  against  NBC  hazards 

Maintain  personal  and  operational  security 

Team-Related: 

Help  peers  and  individual  team  members 

Monitor  peer  and  individual  team  member  performance 

Monitor  team  performance 

Miscellaneous: 

Operate  electronic  equipment 
Prepare  food  and  beverages 
Follow  orders  and  rules 

GWA  Descriptor  Recommendations 

Several  approaches  could  be  used  to  remedy  areas  with  lack  of  coverage.  They  are:  (a) 
add  several  new  specific  GWAs;  (b)  add  only  a  few  broad  GWAs;  or  (c)  expand  the  definitions 
of  existing  GWAs  to  include  the  uncovered  MDs.  We  prefer  adding  new  GWAs  over  modifying 
existing  ones,  because  the  comparability  to  0*NET  would  remain  preserved  if  GWAs  were 
simply  added  and  not  changed,  and  that  comparability  could  serve  the  Army  well  in  the  future. 


60 


Specifically,  we  recommend  adding  one  or  more  Army-specific  GW  As  for  the  Hazard/Combat 
Related  grouping  and/or  for  the  Team-Related  grouping.  If  this  were  undertaken,  then  other 
MDs  having  similar  content  to  these  two  groupings  might  be  identified  to  inform  the 
development  of  such  GW  As.  Conversely,  the  Miscellaneous  grouping  represents  a  hodge-podge: 
“Operate  electronic  equipment”  could  be  its  own  GW  A,  as  perhaps  could  “Prepare  food  and 
beverages.”  “Follow  orders  and  rules”,  however,  does  not  seem  like  an  activity  in  the  sense  of 
the  other  MDs  and  might  best  be  modeled  as  a  work  style  requirement. 

A  Vision  for  the  Future 

Given  what  we  have  learned,  we  are  optimistic  about  the  relatively  rapid  development 
and  implementation  of  an  Army-specific  occupational  analysis  system  that: 

•  incorporates  “as  is”  the  ability  and  skill  domains  of  0*NET 

•  makes  a  limited  number  of  “improvements”  by  adding  key  descriptors  to  the  0*NET 
GWAS  and  Work  Context  domains,  along  the  lines  outlined  above 

•  most  crucially,  includes  a  fully-developed  and  refined  domain  of  Key  Work  Activities, 
organized  into  higher-order  Major  Duties 

•  uses  a  cadre  of  trained  analysts  to  make  the  Ability  and  Skill  ratings  (the  standard 
0*NET  training  given  to  either  experienced  Army  SMEs  or  Army  scientists  would  be 
sufficient) 

•  uses  samples  of  1 5  to  30  SMEs  to  make  the  GWA  and  Work  Context  ratings  for  each 
Army  occupation 

This  system  would  provide  an  extremely  useful  “common  language”  occupational 
analysis  system  for  the  Army  with  strong  links  to  the  civilian  occupational  database.  Such  a  link 
has  obvious  benefits  for  recruitment  and  rapid  mobilization  efforts.  Selection  and  classification 
applications  should  flow  directly  from  such  a  system  as  described  in  Campbell  et  al,  2006  and 
training  and  development  needs  could  be  met  with  a  linkage  of  specific  occupational  tasks  to 
KWAs  and,  hence,  to  Major  Duties,  as  we  briefly  described  in  Chapter  II.  Opportunities  for 
efficiencies  in  training  and  development  applications  would  seem  much  easier  to  identify  with 
such  a  system,  as  would  the  definition  and  development  of  job  performance  criteria  linked  to 
KWAs  and  MDs. 
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Introduction 


The  Occupational  Information  Network  (0*NET)  is  a  comprehensive  system  developed 
by  the  U.S.  Department  of  Labor  that  provides  information  for  812  occupations  within  the  U.S. 
economy.  This  information  is  maintained  in  a  comprehensive  database  which  was  developed  to 
replace  the  Dictionary  of  Occupational  Titles  (DOT)  (U.S.  Department  of  Labor,  1991).  The 
DOT,  which  was  first  published  by  the  Department  of  Labor  in  1939,  provided  information  on 
12,000  occupations.  The  information  was  collected  by  observing  and  interviewing  job 
incumbents,  much  of  which  occurred  during  the  1970s.  In  the  late  1980s,  the  Employment  and 
Training  Administration  (ETA)  of  the  U.S.  Department  of  Labor  embarked  on  a  review  of  the 
DOT.  This  review  was  aimed  at  assessing  the  costs  and  problems  associated  with  maintaining 
the  information.  As  part  of  this  review,  in  1990  the  Secretary  of  Labor  appointed  an  Advisory 
Panel  on  the  DOT  (APDOT).  This  group  recommended  developing  an  electronic  database  to 
house  occupational  information,  collecting  information  on  additional  types  of  job  descriptors, 
and  relying,  primarily,  on  surveys  to  collect  the  requisite  data.  In  addition,  APDOT  noted  the 
importance  of  implementing  timely  updates  and  maintenance  of  the  database.  Based  upon  these 
recommendations,  0*NET  was  created. 

0*NET  Content  Model 

The  0*NET  system  is  based  upon  the  0*NET  Content  Model,  which  provides  a 
framework  for  classifying,  organizing,  and  structuring  0*NET  data.  The  components  of  the 
Content  Model  were  developed  based  on  a  thorough  review  of  an  extensive  body  of  literature 
from  the  job  analysis  arena  within  the  field  of  industrial/organizational  psychology  (e.g., 
Fleishman,  1992;  Lubinski  &  Dawes,  1992).  The  research  team  synthesized  the  available 
information  and  identified  a  taxonomy  of  variables  to  serve  as  the  foundation  of  the  Content 
Model  (Peterson,  et  al.,  1995,  Peterson,  1997,  &  Peterson,  1999).  Since  its  inception,  the  Model 
has  been  reviewed  and  evaluated,  as  a  result,  slightly  revised.  In  general,  the  Content  Model 
contains  four  types  of  descriptors:  job-oriented,  worker-oriented,  cross-occupational,  and 
occupation-specific.  In  turn,  these  descriptors  are  organized  into  the  six  domains  shown  in  Figure 
1  and  described  below. 

The  six  Content  Model  domains  are: 

>  Worker  Characteristics — enduring  characteristics  that  may  influence  both 

performance  and  the  capacity  to  acquire  knowledge  and  skills  required  for  effective 
work  performance.  Included  in  this  domain  are: 

•  Abilities:  enduring  attributes  of  the  individual  that  influence  performance. 

•  Occupational  Interests:  Preferences  for  work  environments  and  outcomes. 
Occupational  Interest  Profiles  (OIPs)  are  compatible  with  Holland's  (1997)  model 
of  personality  types  and  work  environments. 

•  Work  Values:  Global  aspects  of  work  composed  of  specific  needs  that  are 
important  to  a  person's  satisfaction.  Occupational  Reinforcer  Patterns  (ORPs)  are 
based  on  the  Theory  of  Work  Adjustment  (Dawis  &  Lofquist,  1984). 

•  Work  Styles:  personal  characteristics  that  can  affect  how  well  someone  performs 
a  job. 
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Figure  1.  0*NET  Content  Model 

>  Worker  Requirements — work-related  attributes  that  are  acquired  and/or  developed 

through  experience  and  education.  Included  in  this  domain  are: 

•  Basic  Skills:  developed  capacities  that  facilitate  learning  and  information  acquisition. 

•  Cross-functional  Skills:  developed  capacities  that  facilitate  performance  of 
activities  that  occur  across  jobs. 

•  Knowledges:  Organized  sets  of  principles  and  facts  applying  in  general  domains. 

•  Education:  Prior  educational  experience  required  to  perform  in  a  job. 

>  Experience  Requirements — requirements  that  are  explicitly  linked  to  certain  types 

of  work  activities.  This  domain  includes: 

•  Experience  and  Training:  The  amount  of  work  activity  required  in  order  to  be 
hired  to  perform  the  target  job. 

•  Basic  Skills  -  Entry  Requirement:  The  skills  required  to  be  hired  for  a  job;  these 
skills  facilitate  learning  or  the  more  rapid  acquisition  of  knowledge. 

•  Cross-functional  Skills  -  Entry  Requirement:  The  skills  required  to  be  hired  for  a 
job;  these  skills  facilitate  performance  of  activities  that  occur  across  jobs. 

•  Licensing:  Licenses,  certificates,  or  registrations  that  are  awarded  to  show  that  a 
job  holder  has  gained  certain  skills.  This  includes  requirements  for  obtaining  these 
credentials,  and  the  organization  or  agency  requiring  their  possession. 
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>  Occupation  Requirements — variables  or  detailed  elements  that  describe  specific 
occupational  requirements.  Included  in  this  domain  are: 

•  Generalized  Work  Activities:  general  types  of  job  behaviors  occurring  on  multiple 
jobs. 

•  Work  Context:  physical  and  social  factors  that  influence  the  nature  of  work. 

•  Detailed  Work  Activities:  detailed  types  of  job  behaviors  occurring  on  multiple 
jobs. 

•  Organizational  Context:  Characteristics  of  the  organization  that  influence  how 
people  do  their  work. 

>  Workforce  Characteristics — variables  that  define  and  describe  the  general 
characteristics  of  occupations  that  may  influence  occupational  requirements  (labor 
market  information,  occupational  outlook). 

>  Occupation-Specific  Information — variables  or  other  elements  (tasks,  tools  and 
technology)  that  apply  to  a  single  occupation  or  a  narrowly  defined  job  family. 

Within  each  Content  Model  domain,  information  is  organized  by  different  levels  of 
description.  The  hierarchical  structure  of  the  Content  Model  can  be  thought  of  as  a  staircase  that 
leads  0*NET  users  to  the  specific  level  of  worker-  or  job-related  information  needed  for  their 
particular  purpose.  Information  on  277  descriptors  is  gathered  as  part  of  the  0*NET  program  and 
even  more  data  is  collected  by  other  federal  agencies  (e.g.,  Bureau  of  Labor  Statistics). 

Occupational  Structure 

As  noted  above,  the  Content  Model  provides  the  informational  framework  for  an 
occupation.  The  organizational  schema  for  all  occupations  is  provided  by  the  0*NET-S0C 
taxonomy  which  is  based  on  the  Standard  Occupational  Classification  (SOC,  U.S.  Department  of 
Labor,  Bureau  of  Labor  Statistics  (2004).  The  initial  taxonomic  system  was  based  on  an 
Occupational  Employment  Statistics  classification  (0*NET  OU  1 998).  However,  in  2000,  the 
Office  of  Management  and  Budget  (OMB)  required  all  government  agencies  to  collect 
occupational  data  based  on  the  SOC.  As  such,  0*NET  underwent  a  major  revision  which 
resulted  in  the  0*NET  SOC  2000  (Levine,  et.  al.,  2000).  This  change  was  followed  by  another 
update  that  resulted  in  the  0*NET-SOC  2006  (National  Center  for  0*NET  Development, 

2006b).  Additional  modifications  to  the  taxonomy  are  planned  when  New  and  Emerging  (N&E) 
occupations  are  added  (National  Center  for  0*NET  Development,  2006a).  Currently,  research  is 
being  conducted  on  high  growth  industries  to  identify  potential  new  and  emerging  occupations. 
When  this  research  is  complete,  these  additional  occupations  will  be  added  to  the  taxonomy. 

During  the  revision  process,  it  was  paramount  to  identify  a  taxonomy  that  would  permit  the 
collection  of  data  at  an  appropriate  level  of  specificity  and  would  reflect  the  changing  nature  of  work 
in  light  of  new  technologies  and  innovative  business  practices  (National  Center  for  0*NET 
Development,  2006b).  In  a  large  majority  of  the  cases,  the  0*NET  SOC  is  identical  to  an  occupation 
in  the  SOC.  In  other  cases,  the  0*NET-S0C  is  more  detailed  than  the  original  SOC  occupation. 
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There  were  five  types  of  modifications  that  occurred  during  the  2006  revisions  to  the 
taxonomy: 

1 .  Detailed  0*NET-S0C  occupations  were  aggregated  to  the  SOC-level  occupation. 

2.  Two  or  more  detailed  0*NET-S0C  occupations  were  aggregated  into  a  new  detailed 
0*NET-SOC  occupation. 

3.  Detailed  0*NET-S0C  occupations  were  subsumed  by  an  existing  detailed  0*NET- 
SOC  occupation. 

4.  Detailed  0*NET-S0C  occupations  were  subsumed  by  a  SOC-level  occupation. 

5.  Detailed  0*NET-S0C  occupations  were  added  to  the  taxonomy. 

(National  Center  for  0*NET  Development,  2006b) 

As  a  result,  the  new  occupational  taxonomy  includes  949  titles,  812  of  which  represent 
data-level  occupations.  That  is,  the  0*NET  program  is  collecting  updated  data  for  these  812 
occupations.  The  remaining  occupations  include  military  occupations,  “all  other”  occupational 
titles,  and  SOCs  which  are  broken  down  into  more  detail  in  0*NET.  No  data  will  be  collected  on 
these  137  occupations;  they  will  remain  in  the  system  by  title  only. 

For  users  of  previous  classification  systems,  including  the  DOT,  Classification  of 
Instructional  Program  (CIP),  Military  Occupational  Classification  (MOC),  and  Registered 
Apprenticeship  Information  System  (RAIS),  the  National  Crosswalk  Service  Center  has  created  and 
maintains  comprehensive  crosswalks,  which  are  available  at  the  following  link: 
http://www.onetcenter.Org/supplemental.html#ncsc  xwalk.  Crosswalks  between  the  0*NET-S0C 

2000  and  0*NET-SOC  2006  can  be  found  at  http://www.onetcenter.Org/taxonomy.html#listings.  All 
crosswalks  are  straightforward,  effective,  and  current. 

Data  Collection  and  the  0*NET  Database 

The  0*NET  data  collection  program  is  a  continual  process  aimed  at  identifying  and 
maintaining  current  information  on  the  characteristics  of  workers  and  jobs.  The  carefully 
designed  methodology  has  received  clearance  from  the  Office  of  Management  and  Budget 
(OMB),  a  testament  to  the  rigor  and  quality  of  the  work  (for  the  recent  OMB  clearance  packet, 
see  http://www.onetcenter.org/dl  files/omb2005/Supporting  Statement2.pdf). 

The  information  that  populates  the  0*NET  database  is  collected  from  four  primary 
sources:  legacy  analysts,  incumbents,  occupational  experts,  and  analysts.  The  data  collection 
process  can  be  divided  into  three  broad  phases:  analyst,  update,  and  in-demand.  Initially,  ratings 
were  obtained  from  occupational  analysts  using  DOT  information  on  a  set  of  descriptors.  This 
yielded  the  first  database  -  0*NET  98  -  which  contained  1,222  Occupational  Units  that  were 
based  on  BLS  OES  codes.  Then,  as  previously  noted,  the  occupational  taxonomy  was  modified 
to  be  more  consistent  with  the  SOC.  0*NET  4.0  includes  the  ratings  of  the  900+  occupations  in 
the  0*NET-SOC  2000  and  represents  the  final  release  of  “analyst  ratings  only”  data.  This 
database  is  referred  to  as  the  Analyst  database. 

Moving  into  the  Update  Phase,  subsequent  databases  have  included  additional  descriptor 
information  and  have  reflected  input  from  incumbents  as  well  as  analysts.  Beginning  in  June 

2001  and  continuing  to-date,  the  full-scale  data  collection  effort  relies  on  incumbent  participation 
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and  involves  updating  all  occupational  information.  Prior  to  implementing  the  new  procedures, 
the  U.S.  Department  of  Labor  and  the  National  0*NET  Consortium  conducted  extensive 
research  to  identify  the  most  effective  methodology  for  surveying  incumbents.  Of  particular 
interest  was  an  investigation  of  the  impact  alternative  approaches  had  on  response  rates.  The 
resulting  methodology  included  the  following  two  key  features: 

1 .  identifying  a  statistically  random  sample  of  businesses  likely  to  have  employees  in  the 
target  occupations,  and 

2.  selecting  a  random  sample  of  employees  in  those  occupations  within  those  businesses. 

The  targeted  incumbents  provide  ratings  on  occupational  tasks,  skills,  generalized  work 
activities,  knowledge,  education  and  training,  work  styles,  and  work  context  areas.  Clearly, 
asking  each  respondent  to  provide  judgments  on  all  of  these  elements  would  be  quite 
burdensome.  Therefore,  with  the  exception  of  the  task  questionnaire  and  demographic  questions 
which  all  raters  receive,  the  questions  are  divided  among  four  different  questionnaires  which  are 
randomly  assigned  to  the  sampled  incumbents. 

Importance  and  level  information  regarding  the  abilities  associated  with  these 
occupations  is  being  collected  from  trained  analysts  who  rely  on  updated  occupational 
information  provided  by  job  incumbents.  It  should  be  noted  that  there  are  theoretical  or 
philosophical  reasons  for  preferring  one  rater  group  to  the  other  for  collecting  different  types  of 
data.  For  example,  incumbents  are  generally  more  familiar  with  the  day-to-day  duties  of  their 
job;  therefore,  they  are  the  best  source  of  information  regarding  tasks  and  GW  As.  In  contrast,  it 
is  likely  that  trained  analysts  understand  the  ability  constructs  better  than  incumbents  and 
therefore  should  provide  the  ability  data.  Although  not  published  yet,  data  for  the  new  and 
emerging  (N&E)  occupations  are  being  collected  from  occupational  experts. 

The  results  of  the  data  collection  efforts  during  the  Update  Phase  have  yielded  high  quality 
data  from  a  national  sample  of  job  incumbents.  There  is  strong  participation  from  both  businesses 
and  employees,  with  over  70%  and  66%  response  rates,  respectively.  In  addition,  more  than  400 
national  associations  have  endorsed  0*NET.  All  ratings  are  carefully  analyzed  (e.g.,  reliability,  inter¬ 
rater  agreement,  standard  errors  of  the  mean)  in  order  to  evaluate  the  quality  of  the  data.  These  data 
checks  provide  invaluable  information  regarding  the  0*NET  methodology  and  constructs  and 
facilitate  the  continual  review  and  improvement  of  the  system  as  a  whole. 

At  the  completion  of  the  Update  Phase,  the  focus  will  shift  to  collecting  data  from  “in- 
demand”  occupations.  These  include  industries  that  are  economically  critical,  are  projected  to 
add  substantial  numbers  of  new  jobs,  or  are  being  impacted  by  technology  and  innovation. 

The  0*NET  database  contains  the  information  associated  with  the  Content  Model.  It 
should  be  noted  that  there  also  are  supplemental  data  files  available.  These  include: 

>  Detailed  Work  Activities:  statements  that  relate  to  work  content  within  a  Generalized 

Work  Activity. 

>  New  Emerging  Tasks:  tasks  that  were  listed  by  incumbents  during  the  survey  process  as 

being  omitted  from  the  existing  task  list  were  identified  as  emerging  tasks. 

>  Lay  Titles:  a  list  of  alternate  occupational  titles  for  0*NET  occupations  derived  from  job 

incumbents,  transactional  analyses,  and  other  governmental  agencies  (e.g.,  BLS,  Census). 
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>  Related  Occupations:  identified  based  on  a  complex,  comprehensive  algorithmic 
formula,  based  on  relatedness  among  selected  0*NET  characteristics. 

>  Tools  and  Technology  (T2):  provides  information  on  machines,  equipment,  tools,  and 
software  that  workers  may  use  for  optimal  functioning  in  a  high  performance  workplace. 
The  information  is  not  exhaustive. 

Currently,  0*NET  releases  updated  databases  twice  a  year.  The  first  update  based  on  the 
new  data  collection  procedures  (Update  Phase)  was  0*NET  5.0,  which  was  released  in  April 
2003.  This  database  included  updated  and  new  information  for  54  occupations,  based  on  the 
0*NET-S0C  2000.  Specifically,  the  new  data  includes  ratings  of  task  statements  and 
information  on  work  context,  work  styles,  training  and  work  experience,  and  education.  In 
addition,  the  abilities,  work  activities,  knowledge,  skills,  job  zone,  and  work  context  data  were 
updated.  Supplemental  modifications  included  an  update  of  the  emerging  occupational  tasks  and 
the  addition  of  detailed  work  activities  and  tools  and  technology. 

Subsequent  database  releases,  which  occur  regularly  at  six  month  intervals,  include  a 
comprehensive  update  of  approximately  100  occupations  for  each  release.  0*NET  10.0  was  the 
first  database  to  contain  the  0*NET-SOC  2006  taxonomy.  The  most  recent  version,  0*NET 
1 1 .0,  is  the  seventh  update  of  the  database  from  the  Data  Collection  Program  and  it  contains 
updated  data  on  680  occupations. 

The  0*NET  database  is  a  very  flexible  tool.  For  example,  one  can  start  with  a  skill  or 
ability  profile  and  identify  occupations  with  similar  profiles.  As  another  option,  one  may  start 
with  an  occupation  and  search  to  find  other  occupations  with  similar  characteristics.  If  the  user 
enters  a  previous  occupation  code  or  title,  0*NET  OnLine  automatically  redirects  the  user  to  its 
current  equivalent.  The  database  is  structured  according  to  the  Content  Model  and  can  be 
downloaded  as  a  flat  file  from  the  0*NET  Center  (www.onetcenter.org)  or  in  Microsoft  Access, 
Visual  FoxPro,  or  SAS/PC  versions.  Examples  of  specific  data  in  0*NET  include  mean 
importance  and  level-based  ratings  for  various  items  (or  descriptors),  and  text  information  on 
occupational  definitions,  descriptor  definitions,  scale  anchors,  and  task  descriptors.  All 
information  is  available  in  a  series  of  files  or  tables.  The  following  Access  tables  contain  data  for 
the  various  currently  available  0*NET  descriptors: 

•  Abilities 

•  Interests 

•  Work  Values 

•  Work  Styles 

•  Skills 

•  Knowledge 

•  Education,  Training,  and  Experience 

•  Work  Activities 

•  Work  Context 

•  Tasks 

•  Occupation  Data  (definitions) 

•  Job  Zones 

(National  Crosswalk  Service  Center,  2006) 
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Uses  of  0*NET  Data 


Information  contained  within  0*NET  is  used  by  a  wide  audience  that  includes 
businesses,  job-seekers,  educators,  students,  counselors,  and  researchers.  The  0*NET  database 
serves  as  the  foundation  for  0*NET  OnLine,  Career  Exploration  Tools,  and  Code  Connector. 
0*NET  OnLine  is  an  interactive  application  for  exploring  and  searching  occupations.  It  helps 
people  get  a  sense  of  the  type  of  worker  and  job  information  that  is  available  through  0*NET, 
and  provides  a  basic  tool  for  accessing  the  information  directly.  From  that  point,  one  can  request 
customized  information  about  a  particular  occupation  or  search  an  entire  0*NET-S0C. 
Alternatively,  one  can  search  across  occupations.  Job  seekers  may  find  the  Skills  Search 
particularly  useful.  As  noted  previously,  it  is  also  possible  to  convert  other  classifications  to  the 
0*NET-S0C  taxonomy  by  using  the  Crosswalk.  0*NET  OnLine  is  updated  whenever 
information  from  job  incumbents  and  analysts  becomes  available. 

The  0*NET  Career  Exploration  Tools  include  several  computerized  assessments  that 
introduce  users  to  a  range  of  career  options.  These  tools  include  the:  Ability  Profiler,  Interest 
Profiler,  Computerized  Interest  Profiler,  Work  Importance  Locator,  and  Work  Importance 
Profiler.  A  third  0*NET  application  is  the  Code  Connector,  which  greatly  facilitates  the  process 
of  interactive  job  coding. 

In  addition  to  the  0*NET  applications  available  through  0*NET  OnLine,  many 
organizations  use  0*NET  data  to  facilitate  the  design,  implementation,  and  support  of  a  variety  of 
programs  and  systems.  The  U.S.  Department  of  Labor’s  website  contains  a  number  of  examples  of 
how  0*NET  has  been  put  “into  action.”  A  few  examples  of  these  applications  include: 

•  West  Virginia  Rehabilitation  Center  is  using  0*NET  to  help  transitioning 
students  translate  career  dreams  and  “can-do”  outlook  into  real  jobs. 

•  A  new  book  on  unfocused  kids  discusses  how  0*NET  can  help  these  children 

•  Wisconsin  Department  of  Workshop  Development  trains  job  center  staff  across  the 
state  on  0*NET  and  0*NET  OnLine  so  they  can  use  the  available  tools  on  their  job. 

•  Faculty  and  staff  at  Temple  University  Center  for  Professional  Development  in 
Career  and  Technical  Education  Center  incorporate  0*NET  OnLine  in  their 
courses  on  program  planning  and  evaluation,  curriculum  development,  and 
cooperative  education. 

0*NET  has  also  proven  to  be  an  invaluable  source  of  data  when  trying  to  identify  new 
and  emerging  skill  needs.  In  response  to  the  need  for  skilled  3-D  computer  artists  and  traditional 
animators  in  the  multimedia  and  entertainment  industries,  California’s  Employment 
Development  Department  conducted  an  industry  study  using  0*NET’s  survey  data  collection 
instruments  along  with  other  material.  The  results  highlighted  the  gap  between  industry  needs 
and  the  local  labor  market.  In  response,  the  local  training  and  education  initiatives  were 
modified,  in  an  effort  to  close  that  gap. 

0*NET  data  are  also  relevant  for  various  human  capital  management  programs.  For 
example,  ability,  skill,  and  experience  information  could  be  useful  for  employee  selection  purposes, 
whereas  GWA  and  knowledge  data  are  likely  to  be  informative  for  training  purposes.  Furthermore, 
job  evaluation  systems  could  benefit  from  both  GWA  and  work  context  information. 
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Economists  and  educational  professional  may  also  find  relevant  uses  for  0*NET.  For 
example,  0*NET  data  has  been  used  to  investigate  the  vulnerability  of  U.S.  jobs  to  offshoring 
and  to  analyze  the  relationship  between  educational  attainment  and  occupational  competencies 
(e.g.,  knowledge,  skills,  abilities)  (Uhalde,  Strohl,  &  Simkins,  2006). 

0*NET  and  Skills  Demands 

Given  the  characteristics  of  its  occupational  taxonomy  and  Content  Model,  0*NET  is  an 
ideal  source  to  inform  questions  regarding  future  skills  demands.  In  fact,  a  number  of 
organizations  and  states  already  take  advantage  of  0*NET  data  to  project  current  skill  supply 
and  potential  skill  changes.  Of  particular  relevance  is  the  work  done  by  the  Projects  Managing 
Partnership  (PMP)  which  is  a  collaboration  between  (1)  the  U.S.  Department  of  Labor, 
Employment  and  Training  Administration  (ETA);  (2)  the  U.S.  Department  of  Labor,  Bureau  of 
Labor  Statistics  (BLS);  (3)  the  National  Association  of  State  Workforce  Agencies  (NASWA); 
and  (4)  the  State  Projections  Workgroup  ('http://dev.proiectionscentral.comyindex.html).  One 
component  of  this  effort  involves  the  implementation  of  the  Skill  Based  Projections  (SBP)  tool. 
Using  0*NET  data  (skills,  abilities,  and  GWAs)  and  the  target  state/territory’s  short-  and/or 
long-term  occupational  projections,  SBP  tool  identifies  the: 

•  current  skill  supply, 

•  projected  demand,  and 

•  potential  skills  gaps  and  replacement  needs. 

An  example  application  of  this  is  presented  in  Illinois '  Future  Workforce:  Will  There  be  Enough 
Workers  with  the  Right  Skills?  (Ginsburg  &  Robinson,  2006).  In  this  case,  they  found  the  following 
15  job  skills  to  have  the  largest  projected  shortages  for  all  growing  occupations  in  Illinois  in  2012: 

1 .  Reading  comprehension 

2..  Active  listening 

3.  Speaking 

4.  Writing 

5.  Critical  thinking 

6.  Active  learning 

7.  Instructing 

8.  Monitoring 

9.  Coordination 

1 0.  Learning  strategies 

1 1 .  Social  perceptiveness 

12.  Time  management 

13.  Judgment  and  decision-making 

1 4.  Complex  problem  identification 

15.  Mathematics 

The  Geo-Skills  Profile  twww.geoskillanalvzer.com.  2006)  offers  a  slightly  different 
example  of  using  0*NET  to  project  future  skills.  In  this  case  the  0*NET  skills  are  coupled  with 
data  from  BLS  and  the  U.S.  Census  Bureau  to  analyze  workforces,  occupations,  wage  data,  and 
skills  in  labor  markets.  Among  the  information  it  can  generate  is  a  report  that  depicts  the  skills 
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that  drive  the  economy  in  the  targeted  geographic  area,  including  each  skill’s  density  when 
compared  to  the  United  States. 

Given  the  question  about  whether  0*NET  can  provide  information  that  would  be  useful  in 
evaluating  changes  in  skill  demands,  the  short  answer  is  yes.  In  fact,  as  noted  above,  organizations 
are  taking  advantage  of  the  data  associated  with  several  of  the  descriptors  (e.g.,  GW  As,  abilities  and 
skills)  in  the  Content  Model  to  evaluate  the  current  and  projected  demands.  In  addition,  it  seems 
feasible  to  analyze  the  0*NET  data  in  several  other  ways  to  adequately  address  this  issue.  For 
example,  one  could  compare  the  occupational  data  from  the  Analyst  Phase  with  data  from  the  same 
occupations  in  the  Update  Phase.  Recall  that  0*NET  was  initially  populated  with  data  provided  by 
occupational  analysts  using  DOT  information  rather  than  by  incumbents.  In  addition,  a  few 
descriptors  were  added  to  the  Content  Model  since  the  initial  data  collection  phase.  However,  data  on 
the  most  relevant  elements  (i.e.,  skills,  abilities,  and  GW  As)  is  available  at  both  points  in  time.  The 
appeal  of  this  type  of  analysis  is  that  it  allows  for  a  comparison  of  skills  for  the  same  occupation  from 
two  different  time  periods.  One  potential  issue  is  that  the  occupation  taxonomies  differed  during  the 
two  data  collection  efforts,  although  the  crosswalk  between  the  two  occupational  structures  would 
help  overcome  this  issue.  Nevertheless,  a  perfect  one-to-one  comparison  for  all  occupations  is 
impossible.  Another  consideration  when  conducting  this  type  of  analysis  is  that  the  sources  of  data 
are  quite  different.  Although  analysts  provide  the  information  for  abilities  in  both  cases,  incumbents 
provide  skill  and  GWA  data  in  the  updated  database.  Despite  these  limitations,  the  resulting  data 
could  prove  quite  informative. 

Another  set  of  analyses  that  could  be  considered  involves  evaluating  the  emerging  trends 
across  occupations  in  the  current  database  regarding  important  skills/abilities/GWAs.  0*NET 
1 1 .0  contains  680  occupations  and  within  a  few  months  0*NET  12.0  will  be  released  with 
another  100  occupations.  The  occupations  that  are  added  with  each  new  release  of  an  0*NET 
database  tend  to  be  from  a  variety  of  occupational  categories.  Given  this,  these  analyses  could  be 
cut  several  different  ways.  The  skills/abilities/GWAs  can  be  rank-ordered  based  on  importance 
across  all  the  occupations  with  updated  data.  Alternatively,  similar  analyses  can  be  done  by 
major  occupational  category  to  see  if  there  are  different  cross-category  trends.  Since  the  data 
collection  effort  spans  a  number  of  years,  one  could  look  at  annual  trends.  Each  year,  data  is 
collected  on  approximately  200  occupations.  Therefore,  this  run  would  be  comparing  the  200 
occupations  collected  in  2003,  with  those  collected  in  2004,  and  then  2005,  and  so  on. 

Obviously,  one  limitation  of  this  type  of  analysis  is  that  the  nature  of  the  occupations  within  a 
given  year  may  be  quite  different  than  those  in  another  year.  In  addition,  200  occupations  may 
not  be  deemed  sufficient.  Even  if  the  analyses  focusing  across  years  is  abandoned,  it  would  be 
telling  to  conduct  a  close  evaluation  of  the  cross  occupational  trends. 

It  seems  as  though  it  would  also  be  informative  to  take  advantage  of  the  Occupational 
Employment  Survey  (OES)  data  collected  by  BLS.  Since  both  the  0*NET  and  OES  databases 
adopted  the  same  occupational  structure,  one  could  match  information  from  the  two  sources  to 
examine  whether  occupations  requiring  particular  transferable  skills  have  grown.  Also,  as  the 
0*NET  database  continues  to  be  updated,  there  will  be  opportunities  to  conduct  more  relevant 
analyses  (e.g.,  time  series),  the  most  obvious  being  a  comparison  of  the  skills  within  the  same 
occupation  across  time.  Another  analysis  could  involve  using  the  OES  and  0*NET  data  to 
examine  changes  in  occupational  demands  as  well  as  changes  in  the  number  of  individuals 
employed  in  those  occupations. 
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Summary 


0*NET  is  a  rich  source  of  occupational  information  that  is  based  on  a  comprehensive 
Content  Model  containing  a  broad  set  of  descriptors.  The  0*NET  database  is  continually 
updated  with  information  provided  by  a  range  of  workers  from  each  occupation.  Given  the 
wealth  of  0*NET  data,  the  potential  applications  are  endless.  It  appears  as  though  one  of  the 
most  prevalent  uses  for  the  information  is  career  exploration  and  planning.  As  noted,  0*NET  can 
also  inform  many  human  capital  management  programs,  such  as  selection,  promotion,  training 
and  succession  planning.  In  addition,  0*NET  data  and  reports  are  informative  for  educators, 
economists,  sociologists,  as  well  as  other  prominent  fields  in  the  workforce  community. 

It  is  clear  that  there  is  a  wealth  of  information  currently  available  in  0*NET  that  can  be  used 
in  a  variety  of  ways  to  inform  the  current  and  future  skill  demands.  As  noted  above,  0*NET  provides 
skills  data.  However,  one  should  also  be  aware  that  as  Pearlman  (1997)  points  out,  the  term  skills 
seems  to  connote  a  variety  of  different  definitions.  Given  that,  it  may  be  advisable  to  consider  other 
elements  (e.g.,  abilities,  GW  As)  of  the  Content  Model  when  evaluating  the  change  in  skills  demands. 
The  relevance  of  the  element  can  be  determined  by  the  user  based  on  their  0*NET  definition. 

Obviously,  most  informative  analyses  to  address  this  issue  would  revolve  around 
comparing  occupational  data  for  the  same  set  of  occupations  across  time.  It  is  unfortunate  that,  at 
this  point,  the  only  data  available  across  two  time  periods  are  not  entirely  comparable.  Yet,  as 
described  above,  there  are  reasonable  analyses  that  could  be  conducted  to  inform  the  potential 
changes  in  skill  demands.  In  fact,  a  number  of  organizations  are  using  0*NET  data  to  answer 
this  question.  Then  as  the  data  collection  effort  continues,  new  occupational  data  will  be 
available  and  analyses  more  directly  related  to  the  question  can  be  conducted. 
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APPENDIX  B 


MDs  Assigned  to  GWA  Factors 
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Categorization  of  Major  Duties  within  Generalized  Work  Activity  Factors 


GWA  Factor 

Major  Duty  (MD) 

Manual, 

Physical, 

and 

Technical 

Activities 

Working 
With  and 
Directing 
Others 

Working 

with 

Information 

1  Protect  against  NBC  hazards 

X 

2  Handle  demolitions  or  mines 

X 

3  Engage  in  hand-to-hand  combat 

X 

4  Inspect  and  maintain  weapons 

X 

5  Fire  direct  fire  weapons 

X 

6  Troubleshoot  and  repair  weapons 

X 

7  Navigate  from  point  to  point 

X 

X 

8  Maintain  personal  and  operational  security 

X 

9  Provide  emergency  first  aid 

X 

X 

X 

10  Scout  and  identify  targets 

11  Fire  heavy  direct  fire  weapons 

X 

X 

12  Fire  indirect  fire  weapons 

X 

X 

13  Drive  track  vehicles 

X 

14  Install  and  maintain  electronic  equipment 

X 

1 5  Troubleshoot  and  repair  electronic  equipment 

X 

16  Install  and  maintain  electrical  and  power  transmission 
systems 

X 

1 7  Troubleshoot  and  repair  electrical  and  power  transmission 
systems 

X 

1 8  Install  and  maintain  personal  computers  and  peripheral 
equipment 

X 

X 

1 9  Install  and  maintain  computer  networks 

X 

X 

20  Troubleshoot  and  repair  personal  computers  and  computer 
networks 

X 

X 

2 1  Operate  electronic  equipment 

22  Operate  personal  computers  and  networks 

X 

X 

23  Record  and  document  audiovisual  information 

X 

24  Send  and  receive  radio  messages 

X 

25  Collect  and  decode  electronic  signals 

X 

26  Analyze  electronic  signals 

X 

27  Provide  data  processing  and  programming  support 

X 

28  Produce  maps,  overlays,  or  range  cards 

X 
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Categorization  of  Major  Duties  within  Generalized  Work  Activity  Factors 

GWA  Factor 

Major  Duty  (MD) 

Manual, 

Physical, 

and 

Technical 

Activities 

Working 
With  and 
Directing 
Others 

Working 

with 

Information 

29  Provide  technical  direction  and  advice  on  communication 
networks  and  information  systems 

X 

X 

30  Translate  foreign  languages 

X 

3 1  Analyze  intelligence  data 

32  Control  individuals  and  crowds 

X 

X 

X 

33  Collect  information  from  and  on  individuals  and  groups 

34  Inspect  and  maintain  mechanical  equipment 

X 

X 

X 

35  Troubleshoot  and  repair  mechanical  equipment 

X 

36  Operate  gas  and  electric  powered  equipment 

37  Prepare  and  process  forms 

X 

X 

38  Maintain  records  and  files 

X 

39  Write  documents  and  correspondence 

X 

40  Monitor  and  control  financial  resources 

X 

41  Load  and  unload  supplies 

X 

42  Drive  wheeled  vehicles 

X 

43  Drive  water  craft 

X 

44  Inspect,  store,  and  issue  supplies 

X 

X 

45  Order  supplies  and  equipment 

X 

46  Manage  and  control  traffic 

47  Operate  hand-operated  power  excavating  equipment 

X 

X 

48  Operate  heavy  .equipment 

X 

49  Install,  maintain,  and  repair  plastic  and  fiberglass 

X 

50  Repair  metal  structures  or  parts 

X 

5 1  Construct  metal  or  steel  structures 

X 

52  Install,  maintain,  and  repair  pipe  assemblies 

X 

53  Construct  wooden  buildings  and  structures 

X 

54  Construct  masonry  buildings  and  structures 

55  Produce  technical  drawings  and  illustrations 

X 

X 

56  Prepare  food  and  beverages 

X 

57  Prepare  patients  and  equipment  for  medical  procedures 

X 

X 

58  Provide  medical  treatment 

X 

X 

X 

Categorization  of  Major  Duties  within  Generalized  Work  Activity  Factors 


Major  Duty  (MD) 

Manual, 

Physical, 

and 

Technical 

Activities 

GWA  Factor 

Working 
With  and 
Directing 
Others 

Working 

with 

Information 

59 

Schedule  patients  and  medical  services 

X 

X 

60 

Provide  counseling  and  other  interpersonal  interventions 

X 

61 

Perform  laboratory  procedures 

X 

X 

62 

Collect  and  analyze  weather  and  environmental  data 

X 

63 

Conduct  land  surveys 

X 

64 

Deliver  presentations 

X 

X 

65 

Reproduce  printed  materials 

X 

67 

Maintain  physical  fitness 

X 

73 

Follow  orders  and  rules 

X 

74 

Contribute  to  team  tasks 

X 

75 

Direct  peers  and  individual  team  members 

X 

76 

Support  peers  and  individual  team  members 

X 

77 

Train  peers  and  individual  team  members 

X 

78 

Help  peers  and  individual  team  members 

X 

79 

Monitor  peer  and  individual  team  member  performance 

X 

80 

Monitor  team  performance 

X 

81 

Contribute  to  team  planning 

X 

X 

82 

Contribute  to  team  coordination 

X 

83 

Plan  and  organize  operations/missions  and  team  tasks 

X 

X 

84 

Direct  and  motivate  individual  Soldiers 

X 

85 

Train  and  coach  Soldiers 

X 

86 

Communicate  information  to  Soldiers,  peers,  and  superiors 

X 

X 

87 

Administer  personnel  actions  and  procedures 

X 

88 

Manage  and  monitor  operations/missions  and  team  tasks 

X 

89 

Direct  and  lead  platoons/squads/teams 

X 

91 

Support  individual  Soldiers 

X 

92 

Build  and  manage  platoon/squad/team  cohesion 

X 

93 

Engage  and  negotiate  with  host  nationals  and  local  leaders. 

X 

94 

Coordinate  with  other  units  and  non- Army  personnel 

X 

X 

Total  Number  of  MDs 

42 

29 

37 
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Appendix  C 

Major  Duty  and  GWA  Definitions 
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Definition 


Major  Duty 

1  Protect  against  NBC  hazards 

2  Handle  demolitions  or  mines 

3  Engage  in  hand-to-hand  combat 

4  Inspect  and  maintain  weapons 

5  Fire  direct  fire  weapons 

6  Troubleshoot  and  repair  weapons 

7  Navigate  from  point  to  point 

8  Maintain  personal  and 
operational  security 

9  Provide  emergency  first  aid 

10  Scout  and  identify  targets 

11  Fire  heavy  direct  fire  weapons 

12  Fire  indirect  fire  weapons 

13  Drive  track  vehicles 

14  Install  and  maintain  electronic 
equipment 


Uses  protective  clothing,  masks,  and  decontamination  equipment  to  protect 
self,  others,  equipment,  and  supplies  from  nuclear,  biological,  and 
chemical  (NBC)  hazards.  Detects  and  monitors  potential  hazards  using 
NBC  detection  equipment. 

Stores,  places,  charges,  discharges,  and  disarms  explosives,  demolition 
devices,  or  mines. 

Uses  offensive  and  defensive  maneuvers  to  combat  and  protect  self  and 
others  from  hostile  individuals. 

Checks,  disassembles,  assembles,  cleans,  lubricates,  and  adjusts  weapons, 
including  pistols,  rifles,  machine  guns,  hand  grenades,  and  breechblocks. 

Aims,  tracks,  and  fires  individual  weapons  (e.g.,  rifles,  pistols,  machine 
guns,  hand  grenades)  at  designated  targets.  Prepares  and  loads  weapon. 
Clears  and  unloads  weapon.  Arms  and  throws  hand  grenades. 

Finds  the  cause  of  malfunctions  in  weapons  using  technical  manuals,  tools, 
and  test  equipment.  After  the  cause  of  a  problem  in  a  weapon  has  been 
found,  fixes  it  using  the  appropriate  tools  and  necessary  replacement  parts 
by  following  directions  in  the  weapon's  technical  manual. 

Reads  and  interprets  maps,  other  navigational  devices,  and  equipment 
(e.g.,  GPS)  to  locate  position  of  self  and  others.  Determines  grid 
coordinates  and  directions.  Moves  from  point  to  point  in  response  to 
terrain  features  (e.g.,  or  cover  or  concealment)  battle  conditions,  and 
mission,  with  or  without  the  aid  of  maps  and  other  navigational  equipment. 

Selects,  prepares,  and  occupies  individual  tactical  positions  (e.g.,  battle 
positions,  overwatch  positions,  observations  posts),  camouflages  self  and 
equipment,  and  observes  security  procedures. 

Provide  emergency  first  aid  to  individuals  (e.g.,  CPR,  put  on  field  dressing, 
prevent  shock). 

With  or  without  optical  devices  and  other  equipment  (e.g.,  night  sights, 
weapon  sights,  binoculars),  scouts  and  locates  possible  targets  and  their 
position(s).  Identifies  target's  type  (e.g.,  troops,  tanks,  aircraft)  and  intent 
(e.g.,  hostile  or  non-hostile). 

Aims,  tracks,  and  fires  heavy  direct  fire  weapon  (e.g.,  tank  main  guns, 

TOW  missile)  at  targets.  Positions  and  loads  weapons  for  firing.  Unloads 
or  extracts  unused  rounds  or  misfires. 

Aims,  tracks,  and  fires  indirect  weapon  (e.g.,  field  artillery,  heavy  mortars)  by 
adjusting  azimuth  and  elevation  controls  to  hit  designated  targets.  Positions 
and  loads  weapon  for  firing.  Unloads  or  extracts  unused  rounds  or  misfires. 

Drives  track  vehicles  (e.g.,  tank,  APC,  BFV,  etc.)  in  response  to  mission, 
terrain,  and  traffic  controls. 

Installs  and  connects  electronic,  communications,  and  satellite-based 
equipment  (e.g.,  GPS,  radios,  antennas,  satellite  telephones,  radar,  missile 
and  tank  ballistics  computer  systems).  Inspects  and  monitors  equipment  for 
operation.  Conducts  scheduled  services  to  maintain  equipment.  (Does  not 
include  personal  computers  and  computer  networks.) 
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Definition 


Major  Duty 

1 5  Troubleshoot  and  repair 
electronic  equipment 


1 6  Install  and  maintain  electrical 
and  power  transmission  systems 


1 7  Troubleshoot  and  repair 

electrical  and  power  transmission 
systems 


1 8  Install  and  maintain  personal 
computers  and  peripheral 
equipment 

1 9  Install  and  maintain  computer 
networks 

20  Troubleshoot  and  repair  personal 
computers  and  computer 
networks 


2 1  Operate  electronic  equipment 


22  Operate  personal  computers  and 
networks 

23  Record  and  document 
audiovisual  information 

24  Send  and  receive  radio  messages 

25  Collect  and  decode  electronic 
signals 


Troubleshoots  electronic,  communications,  and  satellite-based  equipment 

(e.g.,  GPS,  radios,  antennas,  radar,  missile  and  tank  ballistics  computer 
systems)  to  diagnose  problems  and  malfunctions  using  specialized  test 
equipment  and  manuals.  Repairs  equipment  with  the  appropriate  tools 
(e.g.,  test  sets,  screwdrivers,  pliers,  soldering  guns)  and  necessary 
replacement  parts  by  following  directions  in  the  equipment's  technical 
manual.  (Does  not  include  personal  computers  and  computer  networks.) 

Installs  and  connects  electrical  and  power  transmission  systems  (e.g., 
electrical  wiring,  power  cables,  communications  wiring).  Lays,  splices, 
and  knots  wires  and  cables  using  the  appropriate  tools  (e.g.,  wire  cutters). 
Inspects  and  monitors  systems  for  operation.  Conducts  scheduled  services 
to  maintain  systems. 

Measures  and  tests  electrical  and  power  transmission  system  components  (e.g., 
generators,  wiring  harnesses,  switches,  relays,  circuit  breakers,  wires,  cables) 
to  detect  and  diagnose  problems  and  malfunctions  using  specialized  test 
equipment  and  manuals.  Repairs  system  components  with  the  appropriate  tools 
(e.g.,  wire  strippers,  pliers,  soldering  irons)  and  necessary  replacement  parts  by 
following  directions  in  the  equipment's  technical  manual. 

Connects  personal  computers  and  peripherals,  installs  software,  and 
connects  to  networks.  Monitors  computers  and  peripherals.  Conducts 
scheduled  services  and  upgrades  to  maintain  computers  and  peripherals. 
(Does  not  include  computer  networks.) 

Installs  and  configures  network  hardware  and  software.  Monitors  network 
use  and  performance.  Conducts  scheduled  services  and  upgrades  to 
maintain  network.  (Does  not  include  personal  computers.) 

Troubleshoots  personal  computer  and  network  components  (e.g.,  hard  dnve, 
monitors,  keyboard,  network  router,  network  cables-wiring)  to  detect  and 
diagnose  problems  and  malfunctions  using  specialized  test  equipment  and 
manuals.  Repairs  computer  or  network  components  with  the  appropriate  tools 
(e.g.,  pliers,  screwdrivers,  wrenches)  and  necessary  replacement  parts  by 
following  directions  in  the  equipment's  technical  manual. 

Sets  and  adjusts  the  controls  to  operate  electronic,  communications,  and 
satellite-based  equipment  (e.g.,  GPS,  radios,  antennas,  satellite  telephones, 
radar,  missile  ballistics  controls),  including  tactical  command  and  control 
systems  (e.g.,  Force  XXI  Battle  Command,  Brigade-and-Below  [FBCB2]). 

Operates  and  works  with  personal  computers  and  networks  to  create  and 
edit  documents  and  presentations,  store  and  enter  data  into  databases,  or  to 
search  for  and  process  information.  (Does  not  include  programming). 

Records  and  documents  visual  and  sound  information  for  intelligence 
analysis,  training,  or  documentation  using  audiovisual  equipment  (e.g., 
audio  recorders,  cameras,  videotape,  digital  video). 

Uses  standardized  radio  codes  and  procedures  to  transmit  and  receive 
messages  and  other  information. 

Collects  electronic  signals  and  communications.  Uses  coding  systems  and 
rules  to  decipher  and  interpret  coded  information. 
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Definition 


Major  Duty 

26  Analyze  electronic  signals 

27  Provide  data  processing  and 
programming  support 

28  Produce  maps,  overlays,  or  range 
cards 


29  Provide  technical  guidance  and 
advice  on  the  installation, 
maintenance,  and  use  of 
equipment 

30  Translate  foreign  languages 

3 1  Analyze  intelligence  data 


32  Control  individuals  and  crowds 


33  Collect  information  from  and  on 
individuals  and  groups 

34  Inspect  and  maintain  mechanical 
equipment 

35  Troubleshoot  and  repair 
mechanical  equipment 


36  Operate  gas  and  electric  powered 
equipment 

37  Prepare  and  process  forms 


38  Maintain  records  and  files 

39  Write  documents  and 
correspondence 

40  Monitor  and  control  financial 
resources 


Analyzes  electronic  signals  to  detect  threat  transmitters  and  electronic 
countermeasures. 

Analyzes  data  processing  needs.  Selects  or  prepares,  edits,  tests,  and  runs 
computer  programs.  Documents  process  and  results. 

Uses  drafting,  graphics,  and  related  techniques  to  draw  and  revise  maps  of 
terrain,  including  locations  of  buildings  and  other  objects,  targets,  avenues 
of  approach,  and  maneuver  areas  from  personal  observation  or  available 
materials  (e.g.,  aerial  photographs). 

Explains  and  demonstrates  to  Army  and  non- Army  personnel  at  all  levels 
how  to  install,  maintain,  and  use  equipment.  Answers  technical  questions. 
Provides  expert  advice  to  others  on  issues  related  to  the  installation, 
maintenance,  and  use  of  equipment. 

Translates  written  or  spoken  foreign  language  communications. 

Determines  importance  and  reliability  of  information.  Uses  information  to 
determine  identity,  capabilities,  disposition,  and  movement  of  non-U. S. 
forces  and  personnel. 

Performs  guard  duty,  including  challenge  and  password.  Apprehends  and 
searches  suspected  criminals,  detainees,  or  prisoners.  Guards  and  escorts 
detainees  or  prisoners.  Participates  in  riot  control. 

Collects  and  gathers  information  from  and  on  individuals  and  groups  using 
a  variety  of  techniques  (e.g.,  interviews,  focus  groups,  observations). 

Inspects  and  monitors  mechanical  equipment  (e.g.,  vehicles,  trailers, 
generators,  construction  equipment).  Conducts  scheduled  services  to 
maintain  equipment. 

Troubleshoots  mechanical  system  components  (e.g.,  engines, 
transmissions,  brakes,  hydraulics,  refrigeration  systems)  to  diagnose 
problems  and  malfunctions  using  specialized  test  equipment  and  manuals. 
Repairs  equipment  using  the  appropriate  tools  (e.g.,  wrenches, 
screwdrivers,  gauges,  hammers,  soldering  equipment)  and  necessary 
replacement  parts  by  following  directions  in  the  equipment  manual. 

Operates  gas  and  electric  powered  equipment  (e.g.,  electric  generators,  air 
compressors,  smoke  generators,  quarry  machines,  mobile  washing 
machines,  water  pumps)  to  produce  power  or  process  materials. 

Follows  standardized  procedures  to  prepare  or  complete  forms  and 
documents  (e.g.,  personnel  records,  legal  briefs,  requisition  requests, 
inspection  records).  Obtains  required  authorizations,  as  needed.  Monitors 
and  reviews  forms  for  completeness. 

Collects,  sorts,  indexes,  files,  and  retrieves  records  and  files  (e.g.,  medical 
records,  training  rosters,  personnel  statistics,  supply  inventories,  etc.). 

Prepares  and  writes  letters,  reports,  and  memos.  Proofreads  and  edits 
documents  prior  to  distribution. 

Monitor  and  controls  the  expenditure  of  financial  resources.  Maintains  and 
reviews  accounting  records.  Disperses  and  collects  money  and  money  orders. 


C-4 


41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 


Major  Duty 

Load  and  unload  supplies 


Definition 


Drive  wheeled  vehicles 
Drive  water  craft 
Inspect,  store,  and  issue  supplies 
Order  supplies  and  equipment 
Manage  and  control  traffic 


Operate  hand-operated  power 
excavating  equipment 

Operate  heavy  equipment 


Install,  maintain,  and  repair 
plastic  and  fiberglass 

Repair  metal  structures  or  parts 

Construct  metal  or  steel 
structures 

Install,  maintain,  and  repair  pipe 
assemblies 

Construct  wooden  buildings  and 
structures 

Construct  masonry  buildings  and 
structures 

Produce  technical  drawings  and 
illustrations 

Prepare  food  and  beverages 


Builds  or  assembles  platforms,  cushions,  and  riggings  for  supplies  and 

equipment  to  protect  from  damage  during  transport.  Loads  and  lashes 
materials  onto  transport  vehicles  (land,  sea,  or  air)  to  secure  and  protect 
from  damage  or  loss  during  shipment.  Unpacks  and  unloads  supplies  after 
transport  to  designated  location. 

Drives  wheeled  vehicles  over  roads  and  cross-country  in  response  to 
mission,  terrain,  and  traffic  regulations. 

Drives  water  craft  (e.g.,  boats,  rafts)  in  response  to  mission  requirements 
and  nautical  regulations. 

Inspects  supplies  and  reviews  paperwork  upon  receipt.  Sorts  and  stores 
supplies.  Issues  or  ships  supplies  to  authorized  personnel  or  units. 

Determines  supply  and  equipment  needs  or  evaluates  requests.  Prepares 
and  submits  orders  and  requisitions  for  needed  supplies  and  equipment. 

Manages  and  coordinates  the  departing,  en  route,  arriving,  and  holding  of 
traffic  (land,  air,  or  sea)  by  monitoring  equipment,  communicating  with 
vehicles  and  other  traffic  control  units. 

Uses  hand-operated  power  excavating  equipment  (e.g.,  air  hammers  and 
drills,  paving  breakers,  grinders,  backfill  tampers)  to  build  concrete,  stone, 
or  other  structures. 

Operates  heavy  equipment  (e.g.,  fork  lifts,  cranes,  back-hoes,  and  graders) 
to  load,  unload,  or  move  other  heavy  equipment,  supplies,  construction 
materials  (e.g.,  culvert  pipe,  building  and  bridge  parts)  or  terrain  (e.g., 
earth,  rocks,  trees,  etc.). 

Installs  plastic  or  fiberglass  parts  and  structures.  Fixes  plastic  or  fiberglass 
parts  and  structures  by  cutting,  sawing,  drilling,  sanding,  filling,  gluing, 
and  painting. 

Fixes  metal  structures  or  parts  by  bending,  cutting,  drilling,  welding, 
hammering,  grinding,  soldering,  and  painting. 

Erects  bridges,  communication  antennas,  and  other  steel  structures.  May 
require  the  assistance  of  others  and  use  of  heavy  equipment. 

Installs,  connects,  and  tests  pipe  assemblies  and  fixtures  (e.g.,  plumbing, 
POL  pipelines  and  pumps).  Repairs  pipe  assemblies. 

Measures,  saws,  nails  or  planes  to  frame,  sheathe,  and  roof  buildings,  or 
erects  trestles,  bridges,  and  piers  from  wood. 

Measures,  lays  brick  or  concrete  blocks,  or  builds  forms  and  pours 
concrete  to  construct  walls,  columns,  field  fortifications,  and  other 
concrete  or  masonry  structures. 

Uses  drafting  equipment  or  other  media  (e.g.,  pen,  pencil,  paint)  to  make 
technical  drawings  and  blueprints.  (Does  not  include  maps,  range  cards,  or 
other  field  expedient  drawings.) 

Prepares  food  and  beverages  according  to  recipes  and  meal  plans  (measure, 
mix,  bake).  Inspects  fresh  food  and  staples  for  freshness.  Cleans  equipment 
and  work  area. 
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Major  Duty 

Definition 

57  Prepare  patients  and  equipment 
for  medical  procedures 

Prepares  patients  for  medical  procedures  by  following  prescribed  rules  and 
directions.  Prepares  medical  or  dental  treatment  areas  for  use  by  laying  out 
instruments  and  equipment.  Cleans  equipment  and  area  for  future  use. 

58  Provide  medical  treatment 

Provides  medical  treatment  to  Soldiers  in  the  field  or  in  medical  or  dental 
clinics,  or  administers  veterinary  treatment  to  animals  (e.g.,  administers 
injections,  takes  blood  pressure,  changes  sterile  dressings).  (Does  not 
include  performing  basic  first  aid.) 

59  Schedule  patients  and  medical 
services 

Schedules  and  provides  routine  information  to  persons  seeking  medical, 
dental,  or  counseling  services. 

60  Provide  counseling  and  other 
interpersonal  interventions 

Counsels  individuals  and  groups  (e.g.,  families)  on  personal  issues  and 
relationship  problems  in  a  clinical,  non-supervisory  setting.  (Does  not 
include  coaching  and  counseling  Soldiers  on  performance-based  issues.) 

6 1  Perform  laboratory  procedures 

Prepares  and  handles  samples  for  laboratory  tests  (e.g.,  medical,  chemical, 
biological).  Conducts  various  types  of  laboratory  tests  following 
prescribed  protocols  and  procedures.  Files  and  reports  results. 

62  Collect  and  analyze  weather  and 
environmental  data 

Collects  data  and  information  on  weather  and  environmental  conditions. 
Analyzes  their  effects  on  tactical  operations. 

63  Conduct  land  surveys 

Surveys  terrain  to  determine  elevations,  azimuths,  and  distances  of  terrain 
features.  Records  information. 

64  Deliver  presentations 

Makes  formal  presentations  (e.g.,  briefings,  radio  and  television  broadcasts). 

65  Reproduce  printed  materials 

Reproduces  printed  materials  using  duplicating  machines  (e.g.,  electronic 
copiers,  printing  presses).  Collates  and  binds  materials  using  various  types 
of  bindery  equipment. 

66  Demonstrate  military  presence 

Presents  a  positive  and  professional  image  of  self  and  the  Army  even  when 
off  duty.  Maintains  proper  military  appearance.  Sets  the  precedent  for 
other  Soldiers  to  follow. 

67  Maintain  physical  fitness 

Meets  Army  standards  for  weight,  physical  fitness,  and  strength.  Maintains 
health  (e.g.,  dental  hygiene)  and  fitness  to  meet  requirements,  to  handle  the 
physical  demands  of  the  daily  job,  and  to  endure  the  stress  of  combat. 

68  Manage  own  duties  and 
responsibilities 

Manages  own  responsibilities  (e.g.,  work  assignments,  personal  finances, 
family,  and  personal  well  being),  and  appears  on  duty  prepared  for  work. 

Sets  goals,  makes  plans,  and  critically  evaluates  own  performance.  Works 
effectively  without  direct  supervision  but  seeks  help  when  appropriate. 

69  Demonstrate  extra  effort  and 
personal  initiative  on  the  job 

Demonstrates  high  effort  in  completing  work.  Takes  independent  action 
when  necessary.  Seeks  out  and  willingly  accepts  responsibility  and 
additional  challenging  assignments.  Persists  in  carrying  out  difficult 
assignments  and  responsibilities. 

70  Manage  own  professional 
development 

Develops  job-related  skills,  devoting  time  off-duty  to  study  and  practice 
important  job-related  skills.  Takes  on  additional  job  duties  and 
responsibilities  to  prepare  for  promotion  and  actively  seeks  out 
opportunities  for  self-improvement.  Keeps  up-to-date  technically. 
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71 

72 

73 

74 

75 

76 

77 

78 

79 

80 

81 


Major  Duty 

Definition 

Demonstrate  personal  integrity 

Maintains  high  ethical  standards.  Does  not  succumb  to  peer  pressure  to 

commit  prohibited,  harmful,  or  questionable  acts.  Voluntarily  reports  thefts, 
misconduct,  and  any  other  violations  of  military  order  and  discipline. 
Understands  and  accepts  the  basic  values  of  the  Army  and  acts  accordingly. 

Exhibit  self-control 

Controls  personal  behavior.  Does  not  engage  in  negative  behaviors,  such 
as  alcohol  and  substance  abuse  at  work.  Meets  financial  obligations 
consistently,  displays  emotional  maturity,  and  does  not  allow  personal 
matters  to  interfere  with  professional  duties  and  obligations. 

Follow  orders  and  rules 

Understands  and  carries  out  orders  relayed  orally  or  in  writing.  Displays 
respect  for  authority.  Adheres  to  regulations,  policies,  and  procedures 
while  completing  assignments.  Checks  the  behavior  of  others  to  ensure 
compliance. 

Contribute  to  team  tasks 

Takes  ownership  for  and  completes  assigned  tasks  for  team  according  to 
committed  timelines.  Demonstrates  effort  toward  team  goals.  Does  not 
pass  work  off  to  others  or  take  shortcuts  that  compromise  quality. 

Direct  peers  and  individual  team 
members 

Helps  to  define  goals  and  organize  and  prioritize  tasks  for  peers  and  individual 
team  members.  Generates  plans  and  strategies  for  task  completion,  identifies 
resources  needed  to  meet  team  goals,  and  shares  resources  or  guides  individual 
team  members  to  resources  to  help  complete  their  tasks. 

Support  peers  and  individual 
team  members 

Provides  social  support  and  empathy,  offers  verbal  encouragement  and  acts 
respectfully  towards  peers  and  team  members,  especially  when  tasks  or 
situations  are  difficult  or  demanding.  Facilitates  cohesion  and  effective 
working  relationships  between  team  members  by  acting  honestly, 
communicating  openly  and  helping  to  manage  or  resolve  conflicts.  Does  not 
embarrass  team  members  in  front  of  others,  act  impatiently,  or  blame  others. 

Train  peers  and  individual  team 
members 

Shares  information  with  peers  and  individual  team  members,  provides  task 
explanations  and  demonstrations,  answers  questions,  and  gives  timely  and 
constructive  feedback.  Does  not  withhold  information  about  team-related 
tasks. 

Help  peers  and  individual  team 
members 

Fills  in  or  covers  for  peers  or  individual  team  members  who  are 
overwhelmed  or  absent.  Rearranges  own  schedule  and  demonstrates 
flexibility  to  help  other  peers  or  individual  team  members.  Puts  in  extra 
time  and  effort  to  help  peers  and  team  members  without  being  asked  and 
without  complaining.  Does  not  engage  in  off-task  activities  when  other 
team  members  could  use  help. 

Monitor  peer  and  individual  team 
member  performance 

Observes  and  is  knowledgeable  about  the  performance  of  peers  or  other 
team  members.  Pays  attention  to  what  peers  and  individual  team  members 
are  doing.  Evaluates  progress  of  self  and  others  and  recognizes  when  team 
members  may  need  help. 

Monitor  team  performance 

Pays  attention  to  the  team's  situation,  including  relevant  conditions, 
procedures,  policies,  resources,  systems,  equipment,  technology,  and  level 
of  team  accomplishment.  Notices  and  identifies  team-relevant  problems 
and  obstacles. 

Contribute  to  team  planning 

Helps  in  identifying  alternative  solutions,  strategies,  or  options  for  dealing 
with  problems,  obstacles,  or  decisions.  Helps  in  evaluating  alternative 
courses  of  action,  and  takes  preventive  measures  to  avoid  future  problems. 
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Major  Duty 

82  Contribute  to  team  coordination 


83  Plan  and  organize 
operations/missions  and  team 
tasks 

84  Direct  and  motivate  individual 
Soldiers 


85  Train  and  coach  Soldiers 

86  Communicate  information  to 
Soldiers,  peers,  and  superiors 

87  Administer  personnel  actions  and 
procedures 

88  Manage  and  monitor 
operations/missions  and  team 
tasks 

89  Direct  and  lead 
platoons/squads/teams 


90  Model  correct  behavior  to 
Soldiers 

91  Support  individual  Soldiers 

92  Build  and  manage 
platoon/squad/team  cohesion 

93  Engage  and  negotiate  with  host 
nationals  and  local  leaders. 


94  Coordinate  with  other  units  and 
non-Army  personnel 


Contributes  to  and  encourages  discussion  of  work  distribution,  workload 
balance,  potential  workload  problems,  and  the  sequencing  of  team 
members  activities.  Coordinates  own  task  activities  with  other  team 
members.  Does  not  make  unnecessary  requests  or  overload  other  team 
members. 

Plans  major  operations  or  team  tasks  prior  to  their  actual  execution  in  field 
or  workplace.  Translates  goals  and  objectives  into  tasks  and  activities. 
Forecasts  possible  problems  for  the  platoon/squad/team  and  develops 
strategies  for  addressing  these  problems.  Organizes  and  prioritizes  work. 

Provides  guidance  and  direction  to  individual  Soldiers.  Motivates  Soldiers 
by  providing  them  with  recognition,  encouragement,  constructive 
criticism,  and  other  feedback  as  appropriate.  Helps  to  set  goals  and 
maintains  performance  standards  for  Soldiers.  Monitors  and  counsels 
Soldiers  on  specific  performance  or  personal  problems,  as  needed. 

Trains,  instructs,  and  coaches  Soldiers  on  how  to  complete  technical  job 
tasks.  Assists  Soldiers  in  improving  their  technical  job  skills/proficiency. 

Keeps  Soldiers,  superiors,  and  others  informed  about  factors  and  issues 
affecting  them.  Obtains  and  then  passes  on  information  to  those  who 
should  know. 

Completes  performance  appraisals.  Makes  or  recommends  various 
personnel  actions.  Keeps  and  maintains  adequate  records.  Follows  standard 
operating  procedures. 

Keeps  an  operation  going  once  it  has  been  initiated.  Checks  to  make  sure  that 
Soldiers  are  carrying  out  their  duties  properly.  Makes  sure  they  have  the  right 
equipment.  Monitors  or  evaluates  the  status  of  equipment  readiness. 

Directs  and  leads  platoon/squad/team  activities.  Assigns  NCOs  and  Soldiers 
duties  and  responsibilities  for  completing  platoon/squad/team  tasks. 
Coordinates  the  actions  of  squads/teams  within  unit  and  those  of  individual 
Soldiers. 

Models  the  correct  performance  behavior  to  Soldiers,  whether  it  be 
technical  task  performance  under  adverse  conditions  or  exhibiting 
appropriate  military  bearing. 

Demonstrates  personal  concern  for  Soldiers.  Backs  up  and  supports 
Soldiers  as  appropriate.  Looks  out  for  their  welfare. 

Builds  and  manages  platoon/squad/team  cohesion.  Manages  and  resolves 
internal  conflict  among  team  members.  Promotes  and  sustains  team  morale. 

Interacts  and  meets  with  host  nationals  and  local  leaders  (e.g.,  tribal,  police 
chiefs)  to  obtain  information,  handle  complaints,  settle  disputes,  and 
resolve  grievances  between  and  among  host  nationals.  Negotiates  with  host 
nationals  to  resolve  conflicts  with  U.S.  forces  and  to  secure  their  support 
for  U.S.  miltiary  operations. 

Coordinates  with  other  units  and  non-Army  personnel  (e.g.,  contractors, 
host  nation  forces)  before,  during,  and  after  operations  for  support  and 
equipment.  Shares  information  on  status,  position,  and  actions  with  other 
units  and  non-Army  personnel,  as  needed. 
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GWA  Title 

Definition 

Level  Anchors 

Low 

Moderate 

High 

1 

Getting  Information 

Observing,  receiving,  and 

otherwise  obtaining 
information  from  all 
relevant  sources. 

Follow  a 

standard 

blueprint 

Review  a  budget 

Study 

international 
tax  laws 

2 

Identifying  Objects, 
Actions,  and  Events 

Identifying  information  by 
categorizing,  estimating, 
recognizing  differences  or 
similarities,  and  detecting 
changes  in  circumstances 
or  events. 

Test  an 

automobile 

transmission 

Judge  the 
acceptability 
of  food  products 

Determine  the 
reaction 
of  a  virus  to  a 
new  drug 

3 

Monitoring  Processes, 
Materials,  or 
Surroundings 

Monitoring  and  reviewing 
information  from 
materials,  events,  or  the 
environment,  to  detect  or 
assess  problems. 

Check  to  see 
if  baking 
bread  is  done 

Test  electrical 
circuits 

Check  the  status 
of  a  patient  in 
critical  medical 

care 

4 

Inspecting  Equipment, 
Structures,  or  Materials 

Inspecting  equipment, 
structures,  or  materials  to 
identify  the  cause  of  errors 
or  other  problems  or 
defects. 

Check  that 
doors  to 
building  are 
locked 

Inspect 

equipment  in  a 
chemical 
processing  plant 

Inspect  a 
nuclear  reactor 

5 

Estimating  the 
Quantifiable 
Characteristics  of 
Products,  Events,  or 
Information 

Estimating  sizes, 
distances,  and  quantities; 
or  determining  time,  costs, 
resources,  or  materials 
needed  to  perform  a  work 
activity. 

Estimate  the 
size  of 
household 
furnishings  to 
be  crated 

Estimate  the  time 
required  to 
evacuate  a  city  in 
the  event  of  a 
major  disaster 

Estimate  the 
amount  of 
natural 

resources  that 
lie  beneath  the 
world's  oceans 

6 

Judging  the  Qualities  of 
Objects,  Services,  or 
People 

Assessing  the  value, 
importance,  or  quality  of 
things  or  people. 

Determine 
whether  to 

remove  a  tree 
that  has  been 
damaged 

Determine  the 
value  of  property 
lost  in  a  fire 

Establish  the 
value  of  a 
recently 
discovered 
ancient  art  work 

7 

Evaluating  Information 
to  Determine 

Compliance  with 
Standards 

Using  relevant  information 
and  individual  judgment  to 
determine  whether  events 
or  processes  comply  with 
laws,  regulations,  or 
standards. 

Review  forms 
for 

completeness 

Evaluate  a 
complicated 
insurance  claim 
for  compliance 
with  policy  terms 

Make  a  ruling 
in  court  on  a 
complicated 
motion 

8 

Processing  Information 

Compiling,  coding, 

categorizing,  calculating, 
tabulating,  auditing,  or 
verifying  information  or 
data. 

Tabulate  the 

costs  of 

parcel 

deliveries 

Calculate  the 

adjustments  for 
insurance  claims 

Compile  data 

for  a  complex 
scientific  report 
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GWA  Title 

Definition 

Level  Anchors 

Low 

Moderate 

High 

9 

Analyzing  Data  or 
Information 

Identifying  the  underlying 
principles,  reasons,  or 
facts  of  information  by 
breaking  down 
information  or  data  into 
separate  parts. 

Determine  the 
location  of  a 
lost  order 

Determine  the 
interest  cost  to 
finance  a  new 
building 

Analyze  the 
cost  of  medical 
care  services  for 
all  hospitals  in 
the  country 

10 

Making  Decisions  and 
Solving  Problems 

Analyzing  information  and 
evaluating  results  to 
choose  the  best  solution 
and  solve  problems. 

Determine  the 
meal  selection 
for  a  cafeteria 

Select  the 
location  for  a 
major  department 
store 

Make  the  final 
decision  about  a 
company's  5- 
year  plan 

11 

Thinking  Creatively 

Developing,  designing,  or 
creating  new  applications, 
ideas,  relationships, 
systems,  or  products, 
including  artistic 
contributions. 

Change  the 
spacing  on  a 
printed  report 

Adapt  popular 
music  for  a  high 
school  brand 

Create  new 

computer 

software 

12 

Updating  and  Using 
Relevant  Knowledge 

Keeping  up-to-date 
technically  and  applying 
new  knowledge  to  your 
job. 

Keep  up  with 
price  changes 
in  a  small 
retail  store 

Keep  current  on 
changes  in 
maintenance 
procedures  for 
repairing  sports 
cars 

Lean 

information 
related  to  a 
complex  and 
rapidly 
changing 
technology 

13 

Developing  Objectives 
and  Strategies 

Establishing  long-range 
objectives  and  specifying 
the  strategies  and  actions 
to  achieve  them. 

Plan  the 
holiday 
schedule  for 
an  airline 
workforce 

Develop  the  plan 
to  complete  the 
merger  of  two 
organizations 
over  a  3-year 
period 

Develop  a  10- 
year  business 
plan  for  an 
organization 

14 

Scheduling  Work 
Activities 

Scheduling  events, 
programs,  and  activities, 
as  well  as  the  work  of 
others. 

Make 

appointments 
for  patients 
using  a 

predetermined 

schedule 

Prepare  the  work 
schedule  for 
salesclerks  in  a 
large  retail  store 

Schedule  a 
complex 
conference 
program  with 
multiple, 
parallel  sessions 

15 

Organizing,  Planning, 
and  Prioritizing  Work 

Developing  specific  goals 
and  plans  to  prioritize, 
organize,  and  accomplish 
your  work. 

Organize  a 
work 

schedule  that 
is  repetitive 
and  easy  to 
plan 

Plan  and 
organize  your 
own  activities 
that  often  change 

Prioritize  and 
plan  multiple 
tasks  several 
months  ahead 
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Definition 
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ID 

Low 

Moderate 

High 

16 

Performing  General 
Physical  Activities 

Performing  physical 

activities  that  require 
considerable  use  of  your 
arms  and  legs  and  moving 
your  whole  body,  such  as 
climbing,  lifting, 
balancing,  walking, 
stooping,  and  handling  of 
materials. 

Walk  between 

work  stations 
in  a  small 
office 

Paint  the  outside 

of  a  house 

Climb  up  and 

down  poles  to 

install 

electricity 

17 

Handling  and  Moving 
Objects 

Using  hands  and  arms  in 
handling,  installing, 
positioning,  and  moving 
materials,  and 
manipulating  things. 

Change 
settings  on 
copy 
machines 

Arrange  books  in 
a  library 

Load  boxes  on 
an  assembly 
line 

18 

Controlling  Machines 
and  Processes 

Using  either  control 
mechanisms  or  direct 
physical  activity  to  operate 
machines  or  processes  (not 
including  computers  or 
vehicles). 

Operate  a 
cash  register 

Operate  a  drilling 
ng 

Operate  a 
precision 
milling 
machines 

19 

Working  with 

Computers 

Using  computers  and 
computer  systems 
(including  hardware  and 
software)  to  program, 
write  software,  set  up 
functions,  enter  data,  or 
process  information. 

Enter 

employee 

information 

into  a 

computer 

database 

Write  software 
for  keeping  track 
of  parts  in 
inventory 

Set  up  a  new 
computer 
system  for  a 
large 

multinational 

company 

20 

Operating  Vehicles, 
Mechanized  Devices,  or 
Equipment 

Running,  maneuvering, 
navigating,  or  driving 
vehicles  or  mechanized 
equipment,  such  as 
forklifts,  passenger 
vehicles,  aircraft,  or  water 
craft. 

Drive  a  car 

Drive  an  18- 
wheel  tractor- 
trailer 

Hover  a 
helicopter  in 
strong  wind 

21 

Drafting,  Laying  out, 
and  Specifying 

Technical  Devices, 

Parts,  and  Equipment 

Providing  documentation, 

detailed  instructions, 
drawings,  or  specifications 
to  tell  others  about  how 
devices,  parts,  equipment, 
or  structures  are  to  be 
fabricated,  constructed, 
assembled,  modified, 
maintained,  or  used. 

Specify  the 

lighting  for  a 
work  area 

Specify  the 

furnishings  for  a 
new  school 

Draw  the 

electronic 
circuitry  for  a 
high-speed 
scientific 
computer 
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Low 

Moderate 

High 

22 

Repairing  and 
Maintaining  Mechanical 
Equipment 

Servicing,  repairing, 
adjusting,  and  testing 
machines,  devices,  moving 
parts,  and  equipment  that 
operate  primarily  on  the 
basis  of  mechanical  (not 
electronic)  principles. 

Make  simple, 
external 
adjustments 
to  a  door 
hinge  with 
ordinary  hand 
tools 

Adjust  a 

grandfather  clock 

Overhaul  a 
power  plant 
turbine 

23 

Repairing  and 
Maintaining  Electronic 
Equipment 

Servicing,  repairing, 
calibrating,  regulating, 
fine-tuning,  or  testing 
machines,  devices,  and 
equipment  that  operate 
primarily  on  the  basis  of 
electrical  or  electronic  (not 
mechanical)  principles. 

Use  knobs  to 
adjust  a 
television 
picture 

Make  repairs  by 
removing  and 
replacing  circuit 
boards 

Use  complex 
test  equipment 
to  calibrate 
electronic 
equipment 

24 

Documenting/Recording 

Information 

Entering,  transcribing, 
recording,  storing,  or 
maintaining  information  in 
written  or 

electronic/magnetic  form. 

Record  the 
weights  of 
trucks  that 
use  the 
highways 

Document  the 
results  of  a  crime 

scene 

investigation 

Maintain 
information 
about  the  use  of 
orbiting 
satellites  for 
private  industry 
communications 

25 

Interpreting  the 

Meaning  of  Information 
for  Others 

Translating  or  explaining 
what  information  means 
and  how  it  can  be  used. 

Interpret  a 
blood 
pressure 
reading 

Interpret  how 
foreign  tax  laws 
apply  to  U.S. 
exports 

Interpret  a 
complex 
experiment  in 
physics  for 
general 
audiences 

26 

Communicating  with 
Supervisors,  Peers,  or 
Subordinates 

Providing  information  to 
supervisors,  coworkers, 
and  subordinates  by 
telephone,  in  written  form, 
e-mail,  or  in  person. 

Write  brief 
notes  to 

others 

Report  the  results 
of  a  sales 
meeting  to  a 
supervisor 

Create 
videotaped 
presentation  of 
a  company's 
internal  policies 

27 

Communicating  with 
People  Outside  the 
Organization 

Communicating  with 
people  outside  the 
organization,  representing 
the  organization  to 
customers,  the  public, 
government,  and  other 
external  sources.  The 
information  can  be 
exchanged  in  person,  in 
writing,  or  by  telephone  or 
e-mail. 

Have  little 
contact  with 
people 
outside  the 
organization 

Make  standard 
presentations 
about  available 
services 

Prepare  or 
deliver  press 
releases 
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Definition 
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Low 

Moderate 

High 

28 

Establishing  and 
Maintaining 

Interpersonal 

Relationships 

Developing  constructive 
and  cooperative  working 
relationships  with  others 
and  maintaining  them  over 
time. 

Exchange 

greetings  with 
a  coworker 

Maintain  good 

working 
relationships 
with  almost  all 
coworkers  and 
clients 

Gain 

cooperation 
from  a 
culturally 
diverse  group  of 
executives 
hostile  to  your 
company 

29 

Assisting  and  Caring  for 
Others 

Providing  personal 
assistance,  medical 
attention,  emotional 
support,  or  other  personal 
care  to  others  such  as 
coworkers,  customers,  or 
patients. 

Help  a 
coworker 
complete  an 
assignment 

Assist  a  standard 
traveler  in 
finding  lodging 

Care  for 
seriously 
injured  persons 
in  an  emergency 
room 

30 

Selling  or  Influencing 
Others 

Convincing  others  to  buy 
merchandise/goods  or  to 
otherwise  change  their 
minds  or  actions. 

Convince  a 
coworker  to 
assist  you 
with  an 
assignment 

Deliver  standard 
arguments  or 
sales  pitches  to 
convince  others 
to  buy  popular 
products 

Deliver  major 
sales  campaign 
in  a  new  market 

31 

Resolving  Conflicts  and 
Negotiating  with  Others 

Handling  complaints, 
settling  disputes,  and 
resolving  grievances  and 
conflicts,  or  otherwise 
negotiating  with  others. 

Apologize  to 
a  customer 

who 

complains 
about  waiting 
too  long 

Get  two 
subordinates  to 
agree  about 
vacation 
schedules 

Negotiate  a 
major  labor- 
management 
contract 

32 

Performing  for  or 
Working  Directly  with 
the  Public 

Performing  for  people  or 
dealing  directly  with  the 
public.  This  includes 
serving  customers  in 
restaurants  and  stores,  and 
receiving  clients  or  guests. 

Tend  a 
highway  toll 
booth 

Sell  shoes  in  a 
popular  shoe 
store 

Perform  a 
monologue  on 
national  TV 

33 

Coordinating  the  Work 
and  Activities  of  Others 

Getting  members  of  a 
group  to  work  together  to 
accomplish  tasks. 

Exchange 
information 
during  a  shift 
change 

Organize  the 
cleanup  crew 
after  a  major 
sporting  event 

Act  as  general 
contractor 
building  a  large 
industrial 
complex 

34 

Developing  and 

Building  Teams 

Encouraging  and  building 
mutual  trust,  respect,  and 
cooperation  among  team 
members. 

Encourage 

two 

coworkers  to 
stick  with  a 
tough 
assignment 

Lead  an 

assembly  team  in 
an  automobile 
plant 

Lead  a  large 
team  to  design 
and  build  a  new 
aircraft 
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Moderate 
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35 

Training  and  Teaching 

Others 

Identifying  the  educational 

needs  of  others, 
developing  formal 
educational  or  training 
programs  or  classes,  and 
teaching  or  instructing 
others. 

Give 

coworkers 

brief 

instructions 
on  a  simple 
procedural 
change 

Teach  a  social 
sciences  course 
to  high  school 
students 

Develop  and 
conduct  training 
programs  for  a 
medical  school 

36 

Guiding,  Directing,  and 
Motivating 

Subordinates 

Providing  guidance  and 
direction  to  subordinates, 
including  setting 
performance  standards  and 
monitoring  performance. 

Work 

occasionally 
as  a  backup 
supervisor 

Supervise  a  small 
number  of 
subordinates  in  a 
well-paid 
industry 

Manage  a 
severely 
downsized  unit 

37 

Coaching  and 

Developing  Others 

Identifying  the 
developmental  needs  of 
others  and  coaching, 
mentoring,  or  otherwise 
helping  others  to  improve 
their  knowledge  or  skills. 

Show  a 
coworker  how 
to  operate  a 
piece  of 
equipment 

Provide  on-the- 
job  training  for 
clerical  workers 

Coach  a  college 
athletic  team 

38 

Providing  Consultation 
and  Advice  to  Others 

Providing  guidance  and 
expert  advice  to 
management  or  other 
groups  on  technical, 
systems-,  or  process- 
related  topics. 

Work  in  a 
position  that 
requires  little 
advising  of 
others 

Recommend  a 
new  software 
package  to 
increase 
operational 
efficiency 

Provide  ideas 
for  changing  an 
organization  to 
increase 
profitability 

39 

Performing 

Administrative 

Activities 

Performing  day-to-day 
administrative  tasks  such 
as  maintaining  information 
files  and  processing 
paperwork. 

Complete 
routine 
paperwork  on 
standard 
forms 

Complete  tax 
forms  required  of 
self-employed 
people 

Serve  as  the 
benefits  director 
for  a  large 
computer  sales 
organization 

40 

Staffing  Organizational 
Units 

Recruiting,  interviewing, 
selecting,  hiring,  and 
promoting  employees  in 
an  organization. 

Work  in  a 
position  that 
has  minimal 
staffing 
requirements 

Interview 
candidates  for  a 
sales  position 
and  make  hiring 
recommendations 

Direct  a  large 
recruiting  and 
employment 
program  for  a 
large 

international 

manufacturing 

organization 

41 

Monitoring  and 

Controlling  Resources 

Monitoring  and  controlling 
resources  and  overseeing 
the  spending  of  money. 

Work  as  a 
housekeeper 
responsible 
for  keeping 
track  of  linens 

_ 

Work  as  a  chef 
responsible  for 
ordering  food  for 
the  menu 

Serve  as  a 
financial 
executive  in 
charge  of  a 
large  company's 
budget 
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ID 

Major  Duty 

VO 

r- 

OO 
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<N 

CM 

Csl 

<N 

m 

CM 

m 

SD 

6 

Troubleshoot  and  repair  weapons 

1.90 

.40 

1.20 

.60 

.20 

.20 

1.90 

.10 

.81 

.71 

35 

Troubleshoot  and  repair  mechanical 

equipment 

1.80 

.60 

1.00 

.50 

.40 

.20 

1.90 

.00 

.80 

.67 

34 

Inspect  and  maintain  mechanical 
equipment 

1.90 

.50 

.90 

.60 

.50 

.10 

1.80 

.00 

.79 

.67 

4 

Inspect  and  maintain  weapons 

2.00 

.60 

1.30 

.50 

.10 

.10 

1.60 

.10 

.79 

.70 

17 

Troubleshoot  and  repair  electrical  and 

power  transmission  systems 

1.60 

.50 

1.10 

.70 

.10 

.20 

.40 

1.60 

.78 

.56 

15 

Troubleshoot  and  repair  electronic 

equipment 

1.50 

.40 

1.00 

.60 

.10 

.30 

.00 

2.00 

.74 

.66 

12 

Fire  indirect  fire  weapons 

.30 

1.30 

1.70 

1.40 

.70 

.00 

.30 

.00 

.71 

.63 

2 

Handle  demolitions  or  mines 

.70 

1.00 

1.70 

.60 

.20 

.30 

.60 

.50 

.70 

.44 

47 

Operate  hand-operated  power  excavating 
equipment 

.10 

1.40 

1.30 

1.80 

.90 

.00 

.10 

.00 

.70 

.69 

16 

Install  and  maintain  electrical  and  power 

transmission  systems 

1.10 

.60 

1.20 

.50 

.10 

.40 

.30 

1.30 

.69 

.42 

14 

Install  and  maintain  electronic  eauipment 

1.20 

.50 

1.00 

.50 

.00 

.30 

.00 

1.90 

.68 

.61 

51 

Construct  metal  or  steel  structures 

.20 

1.30 

1.30 

1.30 

.80 

.40 

.10 

.00 

.68 

.53 

20 

Troubleshoot  and  repair  personal 

computers  and  computer  networks 

1.40 

.40 

1.30 

.10 

.00 

.20 

.10 

1.70 

.65 

.65 

11 

Fire  heavy  direct  fire  weapons 

.30 

1.20 

1.50 

1.30 

.60 

.00 

.30 

.00 

.65 

.56 

48 

Operate  heavy  equipment 

.10 

.70 

1.20 

1.50 

1.60 

.00 

.10 

.00 

.65 

.65 

50 

Repair  metal  structures  or  parts 

.70 

1.20 

1.30 

1.10 

.20 

.20 

.20 

.00 

.61 

.49 

18 

Install  and  maintain  personal  computers 
and  peripheral  equipment 

1.20 

.50 

1.20 

.10 

.00 

.20 

.00 

1.60 

.60 

.60 

41 

Load  and  unload  supplies 

.30 

1.90 

1.80 

.10 

.30 

.10 

.10 

.00 

.58 

.74 

49 

Install,  maintain,  and  repair  plastic  and 
fiberglass 

.60 

1.00 

1.30 

.80 

.20 

.20 

.40 

.00 

.56 

.42 

53 

Construct  wooden  buildings  and 
structures 

.30 

1.40 

1.50 

.90 

.20 

.20 

.00 

.00 

.56 

.58 

19 

Install  and  maintain  computer  networks 

.90 

.40 

1.20 

.10 

.00 

.40 

.00 

1.50 

.56 

.54 
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52 

Install,  maintain,  and  repair  pipe 
assemblies 

.90 

1.00 

1.30 

.40 

.20 

.10 

.50 

.00 

.55 

.44 

54 

Construct  masonry  buildings  and 

.30 

1.40 

1.50 

.80 

.10 

.20 

.00 

.00 

.54 

.58 

structures 

36 

Operate  gas  and  electric  powered 
equipment 

.20 

.30 

.70 

1.60 

1.20 

.00 

.10 

.10 

.53 

.55 

1 

Protect  against  NBC  hazards 

.70 

.70 

1.00 

.80 

.20 

.00 

.40 

.30 

.51 

.32 

5 

Fire  direct  fire  weapons 

.10 

.90 

1.40 

1.20 

.20 

.00 

.20 

.00 

.50 

.54 

44 

Inspect,  store,  and  issue  supplies 

1.30 

.70 

1.20 

.00 

.20 

.00 

.00 

.00 

.43 

.53 

65 

Reproduce  printed  materials 

.10 

.40 

1.20 

1.30 

.10 

.30 

.00 

.00 

.43 

.49 

3 

Engage  in  hand-to-hand  combat 

.10 

1.70 

1.00 

.30 

.20 

.00 

.00 

.00 

.41 

.58 

13 

Drive  track  vehicles 

.00 

.20 

.50 

.30 

2.00 

.00 

.10 

.00 

.39 

.63 

42 

Drive  wheeled  vehicles 

.10 

.30 

.40 

.10 

2.00 

.00 

.10 

.00 

.38 

.63 

43 

Drive  water  craft 

.10 

.30 

.40 

.10 

2.00 

.00 

.10 

.00 

.38 

.63 

57 

Prepare  patients  and  equipment  for 
medical  procedures 

.40 

.70 

1.40 

.10 

.00 

.30 

.00 

.00 

.36 

.46 

7 

Navigate  from  point  to  point 

.00 

.80 

.80 

.30 

.60 

.00 

.00 

.10 

.33 

.33 

56 

Prepare  food  and  beverages 

.50 

.50 

1.20 

.30 

.00 

.00 

.00 

.00 

.31 

.40 

58 

Provide  medical  treatment 

.10 

.70 

1.30 

.30 

.00 

.10 

.00 

.00 

.31 

.43 

61 

Perform  laboratory  procedures 

.20 

.30 

1.20 

.50 

.00 

.30 

.00 

.00 

.31 

.38 

9 

Provide  emergency  first  aid 

.10 

1.10 

.90 

.10 

.00 

.10 

.00 

.00 

.29 

.42 

8 

Maintain  personal  and  operational 
security 

.30 

1.00 

.60 

.10 

.00 

.10 

.00 

.00 

.26 

.34 

32 

Control  individuals  and  crowds 

.10 

1.60 

.40 

.00 

.00 

.00 

.00 

.00 

.26 

.52 

67 

Maintain  physical  fitness 

.00 

1.90 

.10 

.00 

.00 

.00 

.00 

.00 

.25 

.62 

21 

Operate  electronic  eauioment 

.10 

.00 

.40 

.70 

.00 

.00 

.10 

.50 

.23 

.25 

Mean  Rating 

.61 

.82 

1.09 

.59 

.39 

.13 

.28 

.32 

SD  of  Means 

.61 

.47 

.39 

.49 

.56 

.13 

.52 

.62 
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APPENDIX  E 


Summary  of  Ratings  of  “Coverage”  By  GWAs  of  Army  Major  Duties  (By 

Major  Duty  Number) 


Summary  of  Ratings  of  [ 'Coverage "  By  GWAs  of  Army  Malory  Duties^  (By  Duty  W  #) 


ID# 

Major  Duty 

Number  of  GWAs  with  Average 
Ratings: 

.50  to  1.00  to 

<0.50  .99  1.50  >1.50 

Avg. 

Ratings 

>.99 

Judged 

Coverage 

1 

Protect  against  NBC  hazards 

4 

3 

1 

0 

1 

Low  Partial 

2 

Handle  demolitions  or  mines 

2 

4 

1 

1 

2 

Full 

3 

Engage  in  hand-to-hand  combat 

6 

0 

1 

1 

2 

Full 

4 

Inspect  and  maintain  weapons 

3 

2 

1 

2 

3 

Full 

5 

Fire  direct  fire  weapons 

5 

1 

2 

0 

2 

Low  Partial 

6 

Troubleshoot  and  repair  weapons 

4 

1 

1 

2 

3 

Full 

7 

Navigate  from  point  to  point 

21 

6 

4 

0 

4 

High  Partial 

8 

Maintain  personal  and  operational  security 

6 

1 

1 

0 

1 

Not 

9 

Provide  emergency  first  aid 

39 

7 

1 

1 

2 

Full 

10 

Scout  and  identify  targets 

14 

6 

1 

2 

3 

Full 

11 

Fire  heavy  direct  fire  weapons 

4 

1 

2 

1 

3 

Full 

12 

Fire  indirect  fire  weapons 

20 

8 

2 

1 

3 

Full 

13 

Drive  track  vehicles 

6 

1 

0 

1 

1 

Full 

14 

Install  and  maintain  electronic  equipment 

3 

2 

2 

1 

3 

Full 

15 

Troubleshoot  and  repair  electronic  equipment 

4 

1 

1 

2 

3 

Full 

16 

Install  and  maintain  electrical  and  power 
transmission  systems 

3 

2 

3 

0 

3 

Low  Partial 

17 

Troubleshoot  and  repair  electrical  and  power 
transmission  systems 

3 

2 

1 

2 

3 

Full 

18 

Install  and  maintain  personal  computers  and 
peripheral  equipment 

20 

6 

4 

1 

5 

Full 

19 

Install  and  maintain  computer  networks 

21 

6 

2 

2 

4 

Full 

20 

Troubleshoot  and  repair  personal  computers  and 
computer  networks 

18 

6 

6 

1 

7 

Full 

21 

Operate  electronic  equipment 

6 

2 

0 

0 

0 

Not 

22 

Operate  personal  computers  and  networks 

17 

3 

2 

1 

3 

Full 

23 

Record  and  document  audiovisual  information 

19 

2 

0 

2 

2 

Full 

24 

Send  and  receive  radio  messages 

17 

3 

3 

0 

3 

High  Partial 

25 

Collect  and  decode  electronic  signals 

13 

5 

2 

3 

5 

Full 

26 

Analyze  electronic  signals 

14 

4 

4 

1 

5 

Full 

27 

Provide  data  processing  and  programming  support 

11 

7 

3 

2 

5 

Full 

28 

Produce  maps,  overlays,  or  range  cards 

12 

10 

1 

0 

1 

High  Partial 

29 

Provide  technical  guidance  and  advice  on  the 
installation,  maintenance,  and  use  of  equipment 

17 

13 

5 

5 

10 

Full 

30 

Translate  foreign  languages 

13 

8 

1 

1 

2 

Full 

E-2 


Summary  of  Ratings  of  " Coverage "  By  GWAs  of  Army  Major  Duties  (By  Duty  ID  #) 


ID# 

Major  Duty 

Number  of  GWAs  with  Average 
Ratings: 

.50  to  1.00  to 

<0.50  .99  1.50  >1.50 

Avg. 

Ratings 

>.99 

Judged 

Coverage 

31 

Analyze  intelligence  data 

8 

9 

5 

1 

6 

Full 

32 

Control  individuals  and  crowds 

20 

3 

1 

1 

2 

Full 

33 

Collect  information  from  and  on  individuals  and 
groups 

25 

10 

3 

2 

5 

Full 

34 

Inspect  and  maintain  mechanical  equipment 

2 

4 

0 

2 

2 

Full 

35 

Troubleshoot  and  repair  mechanical  equipment 

3 

2 

1 

2 

3 

Full 

36 

Operate  gas  and  electric  powered  equipment 

5 

1 

1 

1 

2 

Full 

37 

Prepare  and  process  forms 

15 

3 

3 

2 

5 

Full 

38 

Maintain  records  and  files 

18 

1 

2 

2 

4 

Full 

39 

Write  documents  and  correspondence 

15 

3 

4 

1 

5 

Full 

40 

Monitor  and  control  financial  resources 

8 

7 

6 

2 

8 

Full 

41 

Load  and  unload  supplies 

6 

0 

0 

2 

2 

Full 

42 

Drive  wheeled  vehicles 

7 

0 

0 

1 

1 

Full 

43 

Drive  water  craft 

7 

0 

0 

1 

1 

Full 

44 

Inspect,  store,  and  issue  supplies 

16 

10 

5 

0 

5 

High  Partial 

45 

Order  supplies  and  equipment 

8 

9 

5 

1 

6 

Full 

46 

Manage  and  control  traffic 

10 

10 

3 

0 

3 

High  Partial 

47 

Operate  hand-operated  power  excavating 
equipment 

4 

1 

2 

1 

3 

Full 

48 

Operate  heavy  equipment 

4 

1 

1 

2 

3 

Full 

49 

Install,  maintain,  and  repair  plastic  and 
fiberglass 

4 

2 

2 

0 

2 

Low  Partial 

50 

Repair  metal  structures  or  parts 

4 

1 

3 

0 

3 

Low  Partial 

51 

Construct  metal  or  steel  structures 

4 

1 

3 

0 

3 

Low  Partial 

52 

Install,  maintain,  and  repair  pipe  assemblies 

4 

2 

2 

0 

2 

Low  Partial 

53 

Construct  wooden  buildings  and  structures 

5 

1 

1 

1 

2 

Full 

54 

Construct  masonry  buildings  and  structures 

5 

1 

1 

1 

2 

Full 

55 

Produce  technical  drawings  and  illustrations 

19 

2 

1 

1 

2 

Full 

56 

Prepare  food  and  beverages 

5 

2 

1 

0 

1 

Low  Partial 

57 

Prepare  patients  and  equipment  for  medical 
procedures 

21 

2 

1 

1 

2 

Full 

58 

Provide  medical  treatment 

27 

17 

3 

1 

4 

Full 

59 

Schedule  patients  and  medical  services 

5 

7 

2 

9 

Full 

60 

Provide  counseling  and  other  interpersonal 
interventions 

10 

4 

2 

1 

3 

Full 
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Summary  of  Ratings  of  "Coverage”  By  GWAs  of  Army  Major  Duties^  (By  Duty  ID  #) 


ID# 

Major  Duty 

Number  of  GWAs  with  Average 
Ratings: 

.50  to  1.00  to 

<0.50  .99  1.50  >1.50 

Avg. 

Ratings 

>.99 

Judged 

Coverage 

61 

Perform  laboratory  procedures 

21 

6 

3 

1 

4 

Full 

62 

Collect  and  analyze  weather  and  environmental 
data 

9 

6 

5 

3 

8 

Full 

63 

Conduct  land  surveys 

17 

3 

2 

1 

3 

Full 

64 

Deliver  presentations 

31 

6 

1 

2 

3 

Full 

65 

Reproduce  printed  materials 

6 

0 

2 

0 

2 

Not 

67 

Maintain  physical  fitness 

7 

0 

0 

1 

1 

Full 

73 

Follow  orders  and  rules 

16 

1 

0 

0 

0 

Not 

74 

Contribute  to  team  tasks 

13 

2 

1 

1 

2 

Full 

75 

Direct  peers  and  individual  team  members 

7 

4 

3 

3 

6 

Full 

76 

Support  peers  and  individual  team  members 

12 

1 

1 

3 

4 

Full 

77 

Train  peers  and  individual  team  members 

11 

3 

2 

1 

3 

Full 

78 

Help  peers  and  individual  team  members 

14 

2 

1 

0 

1 

Low  Partial 

79 

Monitor  peer  and  individal  team  member 
performance 

12 

4 

1 

0 

1 

Low  Partial 

80 

Monitor  team  performance 

13 

3 

1 

0 

1 

Low  Partial 

81 

Contribute  to  team  planning 

17 

13 

7 

3 

10 

Full 

82 

Contribute  to  team  coordination 

9 

5 

1 

2 

3 

Full 

83 

Plan  and  organize  operations/missions  and  team 
tasks 

17 

9 

9 

5 

14 

Full 

84 

Direct  and  motivate  individual  Soldiers 

9 

4 

3 

1 

4 

Full 

85 

Train  and  coach  Soldiers 

12 

3 

0 

2 

2 

Full 

86 

Communicate  information  to  Soldiers,  peers, 
and  superiors 

32 

5 

1 

2 

3 

Full 

87 

Administer  personnel  actions  and  procedures 

14 

5 

3 

1 

4 

Full 

88 

Manage  and  monitor  operations/missions  and 
team  tasks 

10 

3 

3 

1 

4 

Full 

89 

Direct  and  lead  platoons/squads/teams 

9 

5 

2 

1 

3 

Full 

91 

Support  individual  Soldiers 

12 

3 

0 

2 

2 

Full 

92 

Build  and  manage  platoon/squad/team  cohesion 

10 

4 

1 

2 

3 

Full 

93 

Engage  and  negotiate  with  host  nationals  and 
local  leaders. 

10 

4 

0 

3 

3 

Full 

94 

Coordinate  with  other  units  and  non- Army 
personnel 

21 

14 

3 

2 

5 

Full 

Note.  Rating  scale  was  0=little  or  no  coverage;  l=partial  coverage;  2=strong  coverage. 
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